{"id":91517,"date":"2022-02-28T11:12:00","date_gmt":"2022-02-28T10:12:00","guid":{"rendered":"https:\/\/blog.wedos.cz\/?p=91517"},"modified":"2022-03-04T10:24:06","modified_gmt":"2022-03-04T09:24:06","slug":"unorove-nocni-ddos-utoky-presahovaly-133-gbps-spickove-300-gbps","status":"publish","type":"post","link":"https:\/\/blog.wedos.com\/cs\/unorove-nocni-ddos-utoky-presahovaly-133-gbps-spickove-300-gbps","title":{"rendered":"\u00danorov\u00e9 no\u010dn\u00ed DDoS \u00fatoky p\u0159esahovaly 133 Gbps, \u0161pi\u010dkov\u011b 300 Gbps"},"content":{"rendered":"\n

V noci z 18.02.2022 na 19.02.2022 jsme zaznamenali doposud nejsiln\u011bj\u0161\u00ed DDoS \u00fatoky na na\u0161\u00ed infrastrukturu od dubna 2021, kdy na n\u00e1s \u0161ly z\u0159ejm\u011b nejsiln\u011bj\u0161\u00ed \u00fatoky v d\u011bjin\u00e1ch \u010desk\u00e9ho internetu.<\/a> Tehdy n\u00e1m to na kr\u00e1tkou chv\u00edli dok\u00e1zalo ucpat t\u0159i 100 Gbps trasy. Tentokr\u00e1t to nebylo tolik o s\u00edle, ale d\u00e9lce, a a\u017e na p\u00e1r z\u00e1kazn\u00edk\u016f \u00fatoky nikdo nezaznamenal.<\/p>\n\n\n\n\n\n\n\n

Prvn\u00ed n\u00e1znak \u00fatoku jsme zaznamenali v p\u00e1tek 18. 2. 2022 po 23:46, kdy na t\u0159ech hlavn\u00edch tras\u00e1ch narostl po\u010det paket\u016f z 340 tis\u00edc za vte\u0159inu na 2,7 milion\u016f paket\u016f za vte\u0159inu (pkt\/s). <\/p>\n\n\n\n

\"\"<\/a>
P\u0159enosy dat na t\u0159ech hlavn\u00edch 100 Gbps tras\u00e1ch v p\u00e1tek 18.02.2022 t\u011bsn\u011b p\u0159ed p\u016flnoc\u00ed.<\/figcaption><\/figure>\n\n\n\n

Ze zp\u011btn\u00e9 anal\u00fdzy dat ze senzoru (cluster server\u016f ur\u010den\u00fdch k detekov\u00e1n\u00ed a anal\u00fdze \u00fatok\u016f) jsme pot\u00e9 zjistili, \u017ee se jednalo o kombinovan\u00fd \u00fatok p\u0159es TCP a ICMP pakety, kter\u00e9 ve \u0161pi\u010dce dos\u00e1hly p\u0159es 3 milion\u016f paket\u016f za vte\u0159inu. Na rozd\u00edl od tras prob\u00edh\u00e1 m\u011b\u0159en\u00ed a vyhodnocov\u00e1n\u00ed jin\u00fdm zp\u016fsobem a zapo\u010d\u00edt\u00e1v\u00e1 se do n\u011bj v\u0161e, co jde p\u0159es s\u00ed\u0165 (v\u010detn\u011b z\u00e1lo\u017en\u00edch tras). Je tak\u00e9 p\u0159esn\u011bj\u0161\u00ed, proto\u017ee vych\u00e1z\u00ed z u\u017e existuj\u00edc\u00edch dat. <\/p>\n\n\n\n

\"\"<\/a>
Anal\u00fdza prn\u00edho \u00fatok\u016f podle dat ze seznzoru.<\/figcaption><\/figure>\n\n\n\n

Po p\u016flnoci se poda\u0159ilo \u00fato\u010dn\u00edk\u016fm na p\u00e1r minut protla\u010dit necel\u00fdch 800 tis\u00edc request\u016f na n\u00e1\u0161 proxy server (p\u0159edt\u00edm mimo DDoS ochrany je i SYN filtr, kter\u00fd blokuje \u00fato\u010d\u00edc\u00ed IP adresy). V podstat\u011b se jednalo o primitivn\u00ed \u00fatok b\u011b\u017en\u00fdmi po\u017eadavky na aplika\u010dn\u00ed vrstv\u011b (layer 7). Pokud n\u00e1\u0161 syst\u00e9m proxy server\u016f zjist\u00ed, \u017ee je pod n\u00e1porem, tak za\u010dne okam\u017eit\u011b vytv\u00e1\u0159et dal\u0161\u00ed proxy servery a rozkl\u00e1dat z\u00e1t\u011b\u017e. V\u00fdhoda vyu\u017e\u00edv\u00e1n\u00ed cloudov\u00e9ho \u0159e\u0161en\u00ed.<\/p>\n\n\n\n

Na kr\u00e1tkou dobu tak pro n\u011bkter\u00e9 z\u00e1kazn\u00edky do\u0161lo k zpomalen\u00ed slu\u017eeb (pokud zrovna jejich web \u0161el p\u0159es proxy servery, kter\u00e9 odbavovaly \u00fatok). Celkov\u011b se v\u0161ak jednalo o zpomalen\u00ed ze stovek ms na zhruba 2 vte\u0159iny na po\u017eadavek.<\/p>\n\n\n\n

Tento \u00fatok trval jen kr\u00e1tkou chv\u00edli a v podstat\u011b se vy\u0159e\u0161il s\u00e1m. Vytvo\u0159ilo se n\u011bkolik proxy server\u016f, kter\u00e9 pomohly s odbaven\u00edm provozu a SYN filtr zaevidoval podez\u0159elou aktivitu nov\u00fdch \u00fato\u010d\u00edc\u00edch IP adres a omezil je.<\/p>\n\n\n\n

Kolegov\u00e9 na podpo\u0159e si samoz\u0159ejm\u011b v\u0161imli, \u017ee na monitoringu se n\u011bco d\u011bje a upozornili na to. Technik, kter\u00fd m\u011bl zrovna slu\u017ebu v\u0161e zkontroloval. Nav\u00edc on-line byla je\u0161t\u011b \u010d\u00e1st lid\u00ed, co maj\u00ed p\u0159\u00edstup z domova. V\u010detn\u011b v\u00fdvoj\u00e1\u0159\u016f. Po men\u0161\u00ed debat\u011b jsme se domn\u00edvali, \u017ee bude klid, ale to nejsiln\u011bj\u0161\u00ed teprve p\u0159i\u0161lo.<\/p>\n\n\n\n

Za\u010dalo to „slab\u0161\u00edm“ \u00fatokem 58,3 Gbps a 6,3 milion\u016f paket\u016f za vte\u0159inu. Takov\u00fdto \u00fatok u\u017e by v\u011bt\u0161in\u011b na\u0161\u00ed konkurence prost\u011b ucpal linku. 100 Gbps linky, kter\u00e9 pou\u017e\u00edv\u00e1me, maj\u00ed jen poskytovatel\u00e9 specializovan\u00fdch slu\u017eeb, kte\u0159\u00ed pot\u0159ebuj\u00ed p\u0159en\u00e1\u0161et velk\u00fd objem dat anebo provozovatel\u00e9 t\u011bch nejv\u011bt\u0161\u00edch datacenter. Na\u0161t\u011bst\u00ed m\u00e1me 3 linky a k tomu je\u0161t\u011b z\u00e1lo\u017en\u00ed trasy.<\/p>\n\n\n\n

\u00dato\u010dn\u00edci to zkou\u0161eli r\u016fzn\u00fdmi druhy \u00fatok\u016f o rozd\u00edln\u00e9 s\u00edle, ale ta prav\u00e1 „chu\u0165ovka“ p\u0159i\u0161la po jedn\u00e9 v noci. Nejsiln\u011bj\u0161\u00ed \u00fatok dos\u00e1hl 133,5 Gbps (jednalo se o n\u011bkolik slo\u017een\u00fdch \u00fatok\u016f). Jedn\u00e1 se o minutov\u00e9 pr\u016fm\u011bry, tak\u017ee strop \u00fatoku byl siln\u011bj\u0161\u00ed, ale na tras\u00e1ch to l\u00e9pe nespo\u010d\u00edt\u00e1me.<\/p>\n\n\n\n

P\u0159edpokl\u00e1d\u00e1me, \u017ee to byl pokus ucpat 100 Gbps linku, kde \u00fato\u010dn\u00edk doufal \u017ee m\u00e1me strop. Tyto \u00fatoky u\u017e byly zn\u00e1t a ne\u017e se s n\u00edm ochrana vypo\u0159\u00e1dala, tak do\u0161lo ke kr\u00e1tk\u00e9mu zpomalen\u00ed slu\u017eeb. \u0158\u00e1dov\u011b b\u011bhem 6 minut (viz. graf n\u00ed\u017ee ve vyhodnocen\u00ed).<\/p>\n\n\n\n

Zaj\u00edmav\u00e9 bylo to, \u017ee kdy\u017e to \u00fato\u010dn\u00edci nezvl\u00e1dli hrubou silou jedn\u00edm siln\u00fdm \u00fatokem, zkou\u0161eli \u010dty\u0159mi del\u0161\u00edmi \u00fatoky, kter\u00e9 p\u0159es\u00e1hly 90,5 Gbps. Tato data jsou pr\u016fm\u011bry a proto \u0161pi\u010dky byly mnohem v\u011bt\u0161\u00ed. <\/p>\n\n\n\n

\"\"<\/a>
\u00datoky z 18.02.2022 na 19.02.2022 podle p\u0159enos\u016f. Graf ze 3 hlavn\u00edch tras.<\/figcaption><\/figure>\n\n\n\n

Co se t\u00fdk\u00e1 po\u010dtu paket\u016f, tak procesory v serverech senzor\u016f i flirt\u016f rozhodn\u011b m\u011bli co po\u010d\u00edtat. Nejsiln\u011bj\u0161\u00ed \u00fatok m\u011bl 13,5 milion\u016f paket\u016f za vte\u0159inu a n\u00e1sleduj\u00edc\u00ed 4 slab\u0161\u00ed 8,7 milion\u016f paket\u016f za vte\u0159inu.<\/p>\n\n\n\n

\"\"<\/a>
\u00datoky z 18.02.2022 na 19.02.2022 podle po\u010dtu paket\u016f. Graf ze 3 hlavn\u00edch tras.<\/figcaption><\/figure>\n\n\n\n

Pak u\u017e byl klid.<\/p>\n\n\n\n

V\u011bt\u0161ina \u00fatok\u016f \u0161la ze zahrani\u010d\u00ed. Ov\u0161em pom\u011brn\u011b velk\u00e1 \u010d\u00e1st i z \u010desk\u00e9ho NIXu, co\u017e mohou b\u00fdt nap\u0159\u00edklad napaden\u00e1 za\u0159\u00edzen\u00ed v \u010cesku. Hodn\u011b spole\u010dnost\u00ed spol\u00e9h\u00e1 na to, \u017ee v p\u0159\u00edpad\u011b nouze vypne zahrani\u010d\u00ed. No m\u00e1me pro v\u00e1s \u0161patnou zpr\u00e1vu. \u00datok o s\u00edle 20 Gbps se d\u00e1 ud\u011blat i z \u010cR. <\/p>\n\n\n\n

\"\"
Grafy p\u0159enos\u016f p\u0159es dodavatele na\u0161\u00ed konektivity spole\u010dnost Kaora z doby \u00fatoku.
Zdroj: https:\/\/www.nix.cz\/ports\/ports\/day<\/figcaption><\/figure>\n\n\n\n

Vyhodnocen\u00ed<\/h2>\n\n\n\n

Po velmi siln\u00fdch \u00fatoc\u00edch z minul\u00e9ho roku jsme upgradovali ochrany. Navy\u0161ovali jsme jak po\u010dty stroj\u016f, tak i p\u0159id\u00e1vali pam\u011b\u0165 a v\u00fdpo\u010detn\u00ed v\u00fdkon do sond a filtr\u016f. Te\u010f to bylo zn\u00e1t. V\u011bt\u0161ina tradi\u010dn\u00edch DDoS \u00fatok\u016f byla detekov\u00e1na do 1 – 3 vte\u0159in a ihned byla probl\u00e9mov\u00e1 konektivita p\u0159esm\u011brov\u00e1na na filtr. M\u00e1lem jsme cel\u00fd \u00fatok ust\u00e1li bez jak\u00fdchkoliv probl\u00e9m\u016f. Na\u0161li jsme i p\u00e1r v\u011bc\u00ed k zlep\u0161en\u00ed.<\/p>\n\n\n\n

Na n\u00e1sleduj\u00edc\u00edm grafu vid\u00edte v\u0161echny po\u017eadavky na webhosting, WMS a WEDOS WebSite. Po p\u016flnoci je tam velk\u00fd n\u00e1r\u016fst request\u016f, co\u017e jsou \u00fatoky na proxy (zn\u00e1zorn\u011bno sv\u011btle zelenou barvou). Jak vid\u00edte nezp\u016fsobilo to \u017e\u00e1dn\u00fd propad. <\/p>\n\n\n\n

Zato velk\u00fd \u00fatok o nam\u011b\u0159en\u00e9 s\u00edle 133,5 Gbps (ve \u0161pi\u010dce p\u0159es 250 Gbps) a 13,5M paket\u016f za sekundu u\u017e ano. Na minutov\u00e9m grafu vid\u00edte \u017ee b\u011bhem celkem 6 minut do\u0161lo k poklesu request\u016f na servery. \u00datok n\u00e1s rozhodn\u011b nevy\u0159adil. Tyto requesty nav\u00edc nemusely b\u00fdt v\u017edy ztracen\u00e9. Pouze pomalej\u0161\u00ed, tak\u017ee mohly dorazit a\u017e v dal\u0161\u00ed minut\u011b. I tak je tam ur\u010dit\u00fd propad zn\u00e1t. Je to ale super v\u00fdsledek na \u00fatok p\u0159esahuj\u00edc\u00ed 100 Gbps ne? \ud83d\ude42<\/p>\n\n\n\n

\"\"<\/a>
Graf v\u0161ech request\u016f z access log\u016f ze v\u0161ech server\u016f, kde je NoLimit, WMS a WEDOS WebSite. Sv\u011btle zelen\u011b je L7 \u00fatok na proxy, \u010derven\u011b pak dopad nejsiln\u011bj\u0161\u00edho \u00fatoku.<\/figcaption><\/figure>\n\n\n\n

P\u016fjde to i bez ztr\u00e1ty request\u016f, ale mus\u00edme to posunout o \u00farove\u0148 v\u00fd\u0161e<\/h2>\n\n\n\n

Abychom dok\u00e1zali podobn\u00e9, ba dokonce o dost siln\u011bj\u0161\u00ed \u00fatoky zvl\u00e1dnout bez v\u011bt\u0161\u00edch dopad\u016f, tak pot\u0159ebujeme na\u0161i ochranu posunout o \u00farove\u0148 v\u00fd\u0161e. K tomu n\u00e1m poslou\u017e\u00ed decentralizovan\u00e1 s\u00ed\u0165 WEDOS Global, kterou za\u010d\u00edn\u00e1me budovat. <\/p>\n\n\n\n

B\u011bhem jara bychom r\u00e1di um\u00edstili 25 serverov\u00fdch sk\u0159\u00edn\u00ed HPE Moonshot 1500 do 25 sv\u011btov\u00fdch datacenter. V ka\u017ed\u00e9m HPE Moonshot 1500 je 45 fyzick\u00fdch server\u016f a 2 switche. Celkem tak po cel\u00e9m sv\u011bt\u011b rozm\u00edst\u00edme 1125 fyzick\u00fdch server\u016f. Tyto servery pak budou odbavovat a filtrovat provoz lok\u00e1ln\u011b a k n\u00e1m do Hlubok\u00e9 u\u017e p\u016fjde v\u0161e p\u0159efiltrovan\u00e9.<\/p>\n\n\n\n

Aktu\u00e1ln\u011b m\u00e1me v provozu ji\u017e 2 tyto boxy a p\u0159ipravujeme rozesl\u00e1n\u00ed a rozm\u00edst\u011bn\u00ed dal\u0161\u00edch.<\/p>\n\n\n\n

M\u00e1me u\u017e podepsan\u00e9 smlouvy, tak\u017ee v b\u0159eznu za\u010dneme pos\u00edlat prvn\u00ed HPE Moonshoty do sv\u011bta. Hodnota hardware, v\u00edcelet\u00fdch smluv a dal\u0161\u00edch v\u011bc\u00ed okolo p\u0159es\u00e1hne 100 milion\u016f K\u010d. Jedn\u00e1 se o nejv\u011bt\u0161\u00ed investici od stavby na\u0161eho druh\u00e9ho priv\u00e1tn\u00edho datacentra WEDOS DC 2.<\/p>\n\n\n\n

Z\u00e1rove\u0148 dokon\u010dujeme novou slu\u017ebu WEDOS Global Protection, kter\u00e1 bude schopn\u00e1 chr\u00e1nit weby na\u0161ich z\u00e1kazn\u00edk\u016f. Aktu\u00e1ln\u011b ji\u017e chr\u00e1n\u00ed vy\u0161\u0161\u00ed des\u00edtky web\u016f p\u0159ed L7 \u00fatoky. Um\u00ed toho ale daleko v\u00edce (r\u016fzn\u00e9 filtrace\/ochrana podle zem\u00ed, roz\u0161\u00ed\u0159en\u00e1 ochrana redak\u010dn\u00edch syst\u00e9m\u016f atd.).<\/p>\n\n\n\n

Vzhledem k aktu\u00e1ln\u00ed situace d\u011bl\u00e1me v\u0161e proto, abychom v\u00fdstavbu WEDOS Global co mo\u017en\u00e1 nejd\u0159\u00edve urychlili. <\/p>\n\n\n\n

Na z\u00e1v\u011br dod\u00e1me, \u017ee brzo p\u0159ineseme \u010dl\u00e1nek, kde bude popsan\u00e9, jak WEDOS Global chr\u00e1n\u00ed web, kter\u00fd je pod \u00fatokem a je na n\u011bj p\u0159es 70 milion\u016f regul\u00e9rn\u00edch p\u0159\u00edstup\u016f za den. Nap\u00ed\u0161eme i o tom, jak WEDOS Global br\u00e1n\u00ed web, na kter\u00fd jde \u00fatok p\u0159es 13 milion\u016f paket\u016f za sekundu. V\u0161e bez sebemen\u0161\u00edho zav\u00e1h\u00e1n\u00ed a web jede a funguje.

Ale to zase p\u0159\u00ed\u0161t\u011b…<\/p>\n","protected":false},"excerpt":{"rendered":"

V noci z 18.02.2022 na 19.02.2022 jsme zaznamenali doposud nejsiln\u011bj\u0161\u00ed DDoS \u00fatoky na na\u0161\u00ed infrastrukturu od dubna 2021, kdy na n\u00e1s \u0161ly z\u0159ejm\u011b nejsiln\u011bj\u0161\u00ed \u00fatoky v d\u011bjin\u00e1ch \u010desk\u00e9ho internetu. Tehdy n\u00e1m to na kr\u00e1tkou chv\u00edli dok\u00e1zalo ucpat t\u0159i 100 Gbps trasy. Tentokr\u00e1t to nebylo tolik o s\u00edle, ale d\u00e9lce, a a\u017e na p\u00e1r z\u00e1kazn\u00edk\u016f \u00fatoky … <\/p>\n

Pokra\u010dovat ve \u010dten\u00ed „\u00danorov\u00e9 no\u010dn\u00ed DDoS \u00fatoky p\u0159esahovaly 133 Gbps, \u0161pi\u010dkov\u011b 300 Gbps“<\/span><\/a><\/p>\n","protected":false},"author":9,"featured_media":91555,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[112],"tags":[122,69,177],"class_list":["post-91517","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bezpecnost","tag-ddos","tag-utoky","tag-wedos-global-protection"],"_links":{"self":[{"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/posts\/91517","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/comments?post=91517"}],"version-history":[{"count":10,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/posts\/91517\/revisions"}],"predecessor-version":[{"id":92837,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/posts\/91517\/revisions\/92837"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/media\/91555"}],"wp:attachment":[{"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/media?parent=91517"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/categories?post=91517"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/tags?post=91517"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}