WEBVTT

00:00.000 --> 00:05.840
This is a weekly summary of interesting news that is relevant for my lectures at

00:05.840 --> 00:11.440
the Eastern Switzerland University of Applied Sciences. My name is Thomas

00:11.440 --> 00:19.440
Bocek and today we will discuss five topics including the Cloudflare outage.

00:19.440 --> 00:28.080
The first topic it's this one here in the application architecture and

00:28.080 --> 00:35.280
distributed systems lecture I talk about scaling and here we have some numbers

00:35.280 --> 00:43.980
what scaling actually means. Google ran 130,000 node Kubernetes cluster in an

00:43.980 --> 00:51.120
experimental mode doubling its previous supported limit of 65,000 nodes due to

00:51.120 --> 00:56.800
scaling demand of AI workloads they mentioned and this shows horizontal

00:56.800 --> 01:03.080
scaling in practice adding more nodes creates new bottlenecks in the control

01:03.080 --> 01:09.160
plane that require solving problems like caching, distributed storage and

01:09.160 --> 01:16.160
efficient scheduling algorithms which we did not fully cover in our lectures yet.

01:16.160 --> 01:23.080
The tests sustained 1,000 pod operations per second and this means the system was

01:23.080 --> 01:28.960
processing 1,000 new pod requests per second while at the same time assigning

01:28.960 --> 01:35.200
1,000 pods to nodes per second and this throughput matters when companies need

01:35.200 --> 01:41.120
to respond to events. If a service suddenly becomes popular and needs to

01:41.120 --> 01:48.240
scale from a few hundred pods to 50,000 pods to handle demand doing this in

01:48.240 --> 01:54.320
under a minute instead of 10 minutes makes a difference. In their tests 39,000

01:54.320 --> 01:59.520
pods were preempted in 93 seconds which means at the same time deleting low

01:59.520 --> 02:04.760
priority pods and creating new high priority ones and another test scaled a

02:04.760 --> 02:12.640
cluster to 130,000 pods in 3 minutes and 40 seconds. They also mentioned that

02:12.640 --> 02:17.880
power constraints are becoming the limiting factor if the pods have for

02:17.880 --> 02:25.360
example an NVIDIA GPU that requires 2 to 3 kilowatts meaning that a 100,000 node

02:25.360 --> 02:31.640
cluster could need hundreds of megawatts and according to the article there were

02:31.640 --> 02:38.760
some key architectural changes for this kind of scaling so the API server they

02:38.760 --> 02:44.480
used a consistent read from cache feature reducing database load by serving

02:44.480 --> 02:52.440
requests from memory and Google uses a proprietary Spanner based key value store

02:52.440 --> 02:58.120
distributed database that spans multiple data centers and can handle 13,000

02:58.120 --> 03:02.880
queries per second for example for lease updates those are periodic heartbeats

03:02.880 --> 03:09.560
proving each node is still alive and healthy and for job management they use

03:09.560 --> 03:16.680
Kueue. Kueue provides batch system capabilities with all or nothing scheduling ensuring

03:16.680 --> 03:23.040
all pods of a job start together or none at all and fair sharing policies that

03:23.040 --> 03:29.320
distribute the resources fairly among competing workloads and while Google

03:29.320 --> 03:38.480
anticipate demand stabilizing around 100,000 nodes the 130,000 node test isn't

03:38.480 --> 03:45.200
officially supported yet and what I also found interesting here they mentioned

03:45.200 --> 03:58.400
that they have customers operating in the ranges of 20 to 65,000 nodes the next

03:58.400 --> 04:07.400
article it's the following article here this article is about event sourcing a

04:07.400 --> 04:17.040
topic that we also discussed in the application architecture lecture and the core idea is

04:17.040 --> 04:23.440
that event sourcing stores state changes as a sequence of immutable events rather

04:23.440 --> 04:28.480
than overwriting data instead of storing just the current account balance the

04:28.480 --> 04:34.920
system records every deposit and withdrawal current state is calculated

04:34.920 --> 04:40.680
by replaying all events and this architecture solves specific distributed

04:40.680 --> 04:46.640
systems problems the append only nature makes writes fast there are no updates

04:46.640 --> 04:53.200
no deletes just inserts and the author mentions in this article that they

04:53.200 --> 05:01.520
process 10,000 events per second using a PostgreSQL server the author provides a

05:01.520 --> 05:07.640
working code examples in Go for the event store including schema design and

05:07.640 --> 05:16.560
concurrency control event sourcing is commonly combined with CQRS which

05:16.560 --> 05:21.960
stands for command query responsibility segregation that separates write

05:21.960 --> 05:28.960
operations from read operations commands modify state and generate events queries

05:28.960 --> 05:35.440
read from optimized projections built from those events this separation allows

05:35.440 --> 05:42.440
each side to scale independently and the challenge is that projections update

05:42.440 --> 05:47.440
asynchronously creating eventual consistency between write and read

05:47.440 --> 05:52.760
models queries may return stale data until projections catch up which is typically

05:52.760 --> 05:59.000
within milliseconds or seconds the next snapshots they are essential for

05:59.000 --> 06:04.560
performance with large event streams rather than replaying thousands or

06:04.560 --> 06:10.280
millions of events every time the system periodically saves snapshots and only

06:10.280 --> 06:18.480
replays events after the snapshot for distributed transactions the saga pattern

06:18.480 --> 06:25.440
coordinates multi-step processes across aggregates with compensation logic to

06:25.440 --> 06:30.840
handle failures if money transfer stops somewhere in the middle of a transaction

06:30.840 --> 06:38.240
then compensating events can reverse earlier steps and also quite interesting

06:38.240 --> 06:43.880
here the lessons learned the trade-offs are clear storage costs increase three

06:43.880 --> 06:49.600
to five times due to storing events projections and snapshots the mental

06:49.600 --> 06:53.640
model also needs a shift but for financial systems requiring complete

06:53.640 --> 06:58.480
audit trails or systems needing time-travel debugging the benefits may

06:58.480 --> 07:08.400
outweigh the costs the next topic is about an expensive mistake it's the

07:08.400 --> 07:14.320
following article here it's not the Cloudflare level expensive mistake which

07:14.320 --> 07:23.400
we will cover later on it's an article about unexpected cloud costs from

07:23.400 --> 07:32.280
network routing misconfiguration the company transferred 20 terabytes of data from

07:32.280 --> 07:38.160
EC2 to S3 in a single day expecting it to be free since both

07:38.160 --> 07:44.040
services were in the same AWS region instead they received a bill for over

07:44.040 --> 07:51.400
$1,000 and the problem when using VPCs with NAT gateways S3

07:51.400 --> 07:57.000
traffic routes through the NAT gateway by default even for same region AWS

07:57.000 --> 08:05.800
services and NAT gateways they charge 4.5 cents per gigabyte for all data

08:05.800 --> 08:11.520
processing regardless of source or destination at 20 terabytes per day this

08:11.520 --> 08:18.800
adds up to approximately $900 daily and the author had researched costs

08:18.800 --> 08:25.600
beforehand and confirmed that EC2 to S3 transfers within the same region are

08:25.600 --> 08:31.680
free this is technically true for the data transfer itself but doesn't account

08:31.680 --> 08:38.600
for NAT gateway processing charges the traffic was routed out through the NAT

08:38.600 --> 08:45.400
gateway and back in incurring processing charges the solution is VPC gateway

08:45.400 --> 08:52.560
endpoints for S3 which create a direct route from the VPC to S3 bypassing the

08:52.560 --> 08:58.920
NAT gateway entirely gateway endpoints for S3 are completely free no hourly

08:58.920 --> 09:06.480
charges the fix required adding a single Terraform resource to create the

09:06.480 --> 09:12.120
endpoint and associate it with route tables and what surprised me is that 20

09:12.120 --> 09:20.280
terabytes through a NAT gateway costs $900 for comparison my 25 gigabits per

09:20.280 --> 09:29.160
second Init7 connection costs 65 Swiss francs around $70 per month less than $3

09:29.160 --> 09:35.200
per day if I fully saturate this connection for 24 hours and the NAT

09:35.200 --> 09:41.040
would be externally reachable which it's not I know but just if we could create a

09:41.040 --> 09:49.840
tunnel into the private subnet I could push approximately 270 terabytes of

09:49.840 --> 09:57.400
traffic to an AWS customer's NAT gateway this would cost me $3 but generate a

09:57.400 --> 10:05.240
bill of over $12,000 for the victim an amplification factor of 4,000 but only

10:05.240 --> 10:08.600
if I can reach the private subnet from outside where I definitely

10:08.600 --> 10:13.640
should not have access and also if I can saturate the 25 gigabits per second

10:13.640 --> 10:17.920
across the internet from a residential connection I expect some

10:17.920 --> 10:22.520
countermeasures from Init7 blocking me or at least contacting me due to this

10:22.520 --> 10:28.080
high unusual traffic in any case I think this is an interesting view on cloud

10:28.080 --> 10:37.880
costs the next article is again about Kubernetes it's the following here the

10:37.880 --> 10:42.480
author describes how Kubernetes can function as a private cloud running on

10:42.480 --> 10:48.560
hardware you control rather than public cloud providers the argument is that

10:48.560 --> 10:53.480
Kubernetes provides the same capabilities as public cloud services

10:53.480 --> 10:59.320
elastic compute storage databases and AI workloads but without vendor lock-in or

10:59.320 --> 11:05.560
unpredictable costs bare-metal Kubernetes can deploy directly on physical servers

11:05.560 --> 11:10.680
without virtualization eliminating hypervisor overhead and this provides

11:10.680 --> 11:15.360
better performance and resource utilization particularly for AI and

11:15.360 --> 11:21.680
high performance workloads and this research here they showed one in five

11:21.680 --> 11:28.480
Kubernetes clusters already deployed on bare-metal and they could observe a

11:28.480 --> 11:34.920
resource utilization improvement of up to 20% and the motivation is cost

11:34.920 --> 11:40.600
control and autonomy public cloud bills can be unpredictable and in the previous

11:40.600 --> 11:45.520
article we saw what can happen with a misconfiguration with bare-metal costs

11:45.520 --> 11:49.480
become predictable you have servers you have racks you have power that you need

11:49.480 --> 11:54.960
to pay you control data location for compliance avoid silent replication to

11:54.960 --> 12:00.960
third-party regions and aren't blocked by provider API limits and the tooling

12:00.960 --> 12:06.400
has also matured infrastructure as code and GitOps handle cluster deployment

12:06.400 --> 12:11.920
Kubernetes operators automate database management queues and observability

12:11.920 --> 12:18.760
without manual intervention and tools like Rook for example provide distributed

12:18.760 --> 12:25.880
storage similar to cloud object storage and CloudNativePG delivers managed

12:25.880 --> 12:32.800
PostgreSQL with automated failover the trade-off is operational complexity you

12:32.800 --> 12:37.040
must handle cluster setup high availability control planes storage

12:37.040 --> 12:42.840
replication and hardware maintenance yourself and the author argues that it

12:42.840 --> 12:47.720
isn't eliminating public cloud but commoditizing it you can still burst

12:47.720 --> 12:53.000
workloads to cloud providers when needed but your default infrastructure is under

12:53.000 --> 12:58.240
your control for organizations hitting cost limits or needing data sovereignty

12:58.240 --> 13:04.480
treating Kubernetes as a private cloud platform offers an alternative to full

13:04.480 --> 13:10.680
public cloud dependence and in my home lab I do not have workloads that require

13:10.680 --> 13:20.360
scaling so I did not yet dive into the complexity of Kubernetes now the

13:20.360 --> 13:28.880
last article is about the Cloudflare outage this was huge last week Cloudflare

13:28.880 --> 13:36.040
experienced a major outage lasting several hours that affected approximately

13:36.040 --> 13:42.400
20% of websites globally the incident took down major platforms including

13:42.400 --> 13:48.720
Twitter Discord ChatGPT and many other services and the outage was triggered by

13:48.720 --> 13:54.360
a database configuration change that caused a feature file for Cloudflare's

13:54.360 --> 14:03.240
Bot Management system to double in size and this resulted in the Bot Management

14:03.240 --> 14:08.320
feature file containing more than 200 features exceeding the pre-allocated

14:08.320 --> 14:15.480
memory limit of 200 and this caused the system to panic and return 500

14:15.480 --> 14:19.800
errors and this file was then distributed to all machines across

14:19.800 --> 14:26.360
Cloudflare's network and the software reading this file had a size limit below

14:26.360 --> 14:30.920
the new size causing it to fail and the file was being regenerated every five

14:30.920 --> 14:35.560
minutes leading to intermittent failures as sometimes good and sometimes bad

14:35.560 --> 14:40.480
configuration files were distributed across the network and the issue was

14:40.480 --> 14:45.600
resolved by stopping the generation of the bad configuration file and manually

14:45.600 --> 14:52.200
inserting a known good file into the distribution system service was largely

14:52.200 --> 14:57.640
restored after about three hours with full restoration taking approximately six

14:57.640 --> 15:03.800
hours and the outage affected core CDN services and many users couldn't access

15:03.800 --> 15:08.920
the Cloudflare dashboard to modify their DNS settings to bypass the proxy

15:08.920 --> 15:16.560
showing a dependency problem many commenters expressed worry about how much

15:16.560 --> 15:20.920
of the internet depends on a single provider when Cloudflare goes down

15:20.920 --> 15:28.120
approximately 20% of websites become unavailable and this centralization was

15:28.120 --> 15:33.560
compared to similar dependencies on AWS and Azure which also experienced

15:33.560 --> 15:40.160
major outages recently and some noted the irony that the internet was originally

15:40.160 --> 15:46.240
designed for decentralization to survive failures but economic forces have pushed

15:46.240 --> 15:52.120
toward centralization multiple commenters pointed out that businesses often

15:52.120 --> 15:58.600
prefer using major providers like Cloudflare because when everyone goes down

15:58.600 --> 16:03.720
together it's easier to explain than being the only one down this creates an

16:03.720 --> 16:08.160
incentive where companies don't invest in redundancy or alternative providers

16:08.160 --> 16:13.200
the phrase nobody gets fired for buying Cloudflare appeared in several

16:13.200 --> 16:19.960
variations and a common frustration was that Cloudflare's status page took 15 to

16:19.960 --> 16:24.200
20 minutes to reflect the outage initially showing everything as

16:24.200 --> 16:29.720
operational this reflects a broader industry problem where status pages are

16:29.720 --> 16:36.640
manually updated rather than automated often for political or SLA reasons also

16:36.640 --> 16:41.600
both the recent AWS and Cloudflare outages share similarities as human

16:41.600 --> 16:46.760
triggered configuration errors but differ in their technical nature and

16:46.760 --> 16:53.560
impact patterns both show the difficulty of managing distributed systems at scale where

16:53.560 --> 17:00.440
changes in one component can have unexpected global effects that's it from

17:00.440 --> 17:06.720
my side this time no crypto news this gives the crypto market time to recover

17:06.720 --> 17:10.200
see you next week