Informa\u010dn\u00ed zpr\u00e1va o technick\u00fdch probl\u00e9mech v na\u0161em datacentru v odpoledn\u00edch hodin\u00e1ch dne 23 a v rann\u00edch hodin\u00e1ch dne 24. kv\u011btna 2014.<\/p>\n\n\n\n\n\n\n\n
V\u0161em na\u0161im z\u00e1kazn\u00edk\u016fm se chceme nejd\u0159\u00edve touto cestou omluvit za v\u00fdpadek slu\u017eeb, kter\u00fd nastal v p\u00e1tek odpoledne 23. kv\u011btna a v sobotu dopoledne 24. kv\u011btna, kdy do\u0161lo k technick\u00fdm probl\u00e9m\u016fm, kter\u00e9 ve v\u00fdsledku vedly k p\u0159eru\u0161en\u00ed nap\u00e1jen\u00ed server\u016f.<\/p>\n\n\n\n
P\u0159eb\u00edr\u00e1me proto ve\u0161kerou zodpov\u011bdnost a nebudeme se vymlouvat na nep\u0159\u00edze\u0148 po\u010das\u00ed nebo vy\u0161\u0161\u00ed moc \u010di souhru n\u00e1hod. Vina je pln\u011b na na\u0161\u00ed stran\u011b a m\u011bli jsme p\u0159edpokl\u00e1dat, \u017ee i k takov\u00e9to situaci m\u016f\u017ee doj\u00edt.<\/p>\n\n\n\n
Z\u00e1rove\u0148 se omlouv\u00e1me za prvn\u00ed p\u00e1te\u010dn\u00ed zpr\u00e1vu, kter\u00e1 V\u00e1s myln\u011b informovala o tom, \u017ee v\u0161e ji\u017e m\u00e1me pod kontrolou a probl\u00e9m je vy\u0159e\u0161en. Nebyl v tom \u017e\u00e1dn\u00fd zl\u00fd \u00famysl, necht\u011bli jsme nikomu lh\u00e1t. Prvn\u00ed t\u0159\u00edminutov\u00fd v\u00fdpadek se n\u00e1m poda\u0159ilo dostat pod kontrolu a ve chv\u00edli, kdy jsme zpr\u00e1vu uve\u0159ejnili, fungovalo v\u00edce ne\u017e 95% v\u0161ech na\u0161ich slu\u017eeb. Na ty ostatn\u00ed nefunk\u010dn\u00ed jsme p\u0159ich\u00e1zeli pozd\u011bji. Omlouv\u00e1me se za to.<\/p>\n\n\n\n
V\u0161e, \u010d\u00edm na\u0161e firma tento p\u00e1tek a sobotu pro\u0161la, je obrovsk\u00fdm ponau\u010den\u00edm. Z\u00e1lo\u017en\u00ed zdroje budeme m\u00edt nyn\u00ed pod zost\u0159en\u00fdm dohledem. Okam\u017eit\u011b zv\u00fd\u0161\u00edme dobu testov\u00e1n\u00ed z\u00e1lo\u017en\u00edch zdroj\u016f, v pond\u011bl\u00ed objedn\u00e1v\u00e1me novou UPS a dal\u0161\u00ed pot\u0159ebn\u00e1 opat\u0159en\u00ed pro lep\u0161\u00ed zabezpe\u010den\u00ed chodu na\u0161ich slu\u017eeb budou n\u00e1sledovat a my V\u00e1s o nich budeme, jako v\u017edy, pravideln\u011b informovat.<\/p>\n\n\n\n
Je\u0161t\u011b jednou se tedy velice omlouv\u00e1me a nyn\u00ed si dovol\u00edme rozepsat podrobn\u011b cel\u00e9 dva dny, abyste m\u011bli p\u0159esnou p\u0159edstavu o tom, co se u n\u00e1s d\u011blo. Popravd\u011b \u0159e\u010deno do\u0161lo k neuv\u011b\u0159iteln\u00fdm shod\u00e1m n\u00e1hod, kter\u00e9 vedly k poru\u0161e n\u011bkolika r\u016fzn\u00fdch syst\u00e9m\u016f a to i p\u0159es to, \u017ee se jedn\u00e1 o nov\u00e1 za\u0159\u00edzen\u00ed v z\u00e1lohovan\u00e9m (redundantn\u00edm) proveden\u00ed.<\/p>\n\n\n\n
Cel\u00e1 situace vznikla shodou v\u00edce navazuj\u00edc\u00edch probl\u00e9m\u016f. Prvn\u00edm d\u016fvodem bylo vedro, kter\u00e9 bylo zdrojem bou\u0159ek a ty zp\u016fsobily zna\u010dn\u00e9 kol\u00eds\u00e1n\u00ed elekt\u0159iny v s\u00edti. Kv\u016fli kol\u00eds\u00e1n\u00ed el. energie do\u0161lo po spu\u0161t\u011bn\u00ed klimatizac\u00ed k po\u0161kozen\u00ed jedn\u00e9 z klimatizac\u00ed a t\u00edm k shozen\u00ed hlavn\u00edho jisti\u010de cel\u00e9ho datacentra. T\u00edm, \u017ee bylo v s\u00edti p\u0159ep\u011bt\u00ed, do\u0161lo k p\u0159eh\u0159\u00e1t\u00ed tohoto hlavn\u00edho jisti\u010de. Automaticky tedy nasko\u010dil dieselov\u00fd gener\u00e1tor, kter\u00fd ov\u0161em po 15 minut\u00e1ch p\u0159estal fungovat kv\u016fli poru\u0161e chlad\u00edc\u00edho syst\u00e9mu motorgener\u00e1toru (posledn\u00ed kontrolu jsme prov\u00e1d\u011bli ve \u010dtvrtek, tedy den p\u0159ed t\u00edm – \u010din\u00edme tak pravideln\u011b ka\u017ed\u00fd t\u00fdden a p\u0159i kontrole gener\u00e1tor nevykazoval \u017e\u00e1dn\u00e9 zn\u00e1mky poruchy), nasko\u010dily tedy UPS, kter\u00e9 dr\u017eely servery v chodu dal\u0161\u00edch 33 minut. V t\u00e9to chv\u00edli jsme ji\u017e v kontaktu s dodavatelem elekt\u0159iny, kter\u00fd k n\u00e1m pos\u00edl\u00e1 oprav\u00e1\u0159e. Po celou dobu fungovala z\u00e1lo\u017en\u00ed klimatizace. Ne\u017e jsme sta\u010dili po\u0161kozen\u00fd jisti\u010d nahradit nouzov\u00fdm \u0159e\u0161en\u00edm, do\u0161lo k vybit\u00ed bateri\u00ed, co\u017e zp\u016fsobilo kompletn\u00ed v\u00fdpadek nap\u00e1jen\u00ed na dobu 3 minut. Pot\u00e9 za\u010dalo fungovat t\u00e9m\u011b\u0159 v\u0161e, ale jak jsme ji\u017e uvedli v\u00fd\u0161e, informovali jsme na na\u0161ich str\u00e1nk\u00e1ch o tom, \u017ee v\u0161e je pod kontrolou, ov\u0161em jednalo se o mylnou informaci, proto\u017ee mal\u00e1 \u010d\u00e1st na\u0161ich slu\u017eeb je\u0161t\u011b funk\u010dn\u00ed nebyla.<\/p>\n\n\n\n
Po cca hodin\u011b pokra\u010dovaly dal\u0161\u00ed v\u00fdkyvy na elektrick\u00e9 s\u00edti, a proto do\u0161lo k po\u0161kozen\u00ed UPS na jedn\u00e9 nap\u00e1jec\u00ed v\u011btvi. A t\u00edm op\u011bt do\u0161lo ke zkratu v na\u0161\u00ed elektrick\u00e9 s\u00edti. Bylo nutn\u00e9 nahradit nouzov\u00e9 \u0159e\u0161en\u00ed hlavn\u00edho jisti\u010de nov\u00fdm pln\u011b funk\u010dn\u00edm hlavn\u00edm jisti\u010dem, kter\u00fd jsme mezi t\u00edm zajistili. Bohu\u017eel UPS z jedn\u00e9 nap\u00e1jec\u00ed v\u011btve byla natolik po\u0161kozena, \u017ee nebylo mo\u017en\u00e9 ji pou\u017e\u00edt a druh\u00e1 UPS m\u011bla natolik vybit\u00e9 baterie z p\u0159edch\u00e1zej\u00edc\u00edho v\u00fdpadku elektrick\u00e9 energie, \u017ee v\u0161e zkolabovalo podruh\u00e9. Tentokr\u00e1t na 13 minut. Po t\u00e9to dob\u011b se poda\u0159ilo hlavn\u00ed jisti\u010d vym\u011bnit a t\u00edm obnovit dod\u00e1vku elekt\u0159iny.<\/p>\n\n\n\n
V\u011bt\u0161ina server\u016f po tomto druh\u00e9m v\u00fdpadku okam\u017eit\u011b nastartovala a fungovala, pouze n\u011bkolik jednotek procent server\u016f (bohu\u017eel je to n\u011bkolik tis\u00edcovek z\u00e1kazn\u00edk\u016f) m\u011blo probl\u00e9my del\u0161\u00ed dobu a jeden mailserver se n\u00e1m poda\u0159ilo obnovit a\u017e v sobotu nad r\u00e1nem.<\/p>\n\n\n\n
Cel\u00e1 firma od p\u00e1te\u010dn\u00edho odpoledne intenzivn\u011b pracovala na odstran\u011bn\u00ed n\u00e1sledk\u016f v\u00fdpadku. Bohu\u017eel cel\u00e1 situace byla o to komplikovan\u011bj\u0161\u00ed, \u017ee v d\u016fsledku v\u00fdpadku do\u0161lo k po\u0161kozen\u00ed prim\u00e1rn\u00edho i z\u00e1lo\u017en\u00edho firewallu na\u0161ich kancel\u00e1\u0159\u00ed a t\u00edm jsme nem\u011bli p\u0159\u00edstup k internetu z na\u0161ich kancel\u00e1\u0159\u00ed a k server\u016fm, ke kter\u00fdm je z bezpe\u010dnostn\u00edch d\u016fvod\u016f povolen p\u0159\u00edstup pouze z po\u010d\u00edta\u010d\u016f v na\u0161ich kancel\u00e1\u0159\u00edch. T\u00edm se obnova zb\u00fdvaj\u00edc\u00edch nefunk\u010dn\u00edch slu\u017eeb Jednalo se o n\u011bkolik m\u00e1lo kus\u016f serer\u016f) velmi zpomalila a komplikovala. Museli jsme zajistit n\u00e1hradn\u00ed (t\u0159et\u00ed) firewall a po jeho zprovozn\u011bn\u00ed jsme postupn\u011b v\u0161echny slu\u017eby obnovili.<\/p>\n\n\n\n
Vzhledem k po\u0161kozen\u00ed UPS na jedn\u00e9 nap\u00e1jec\u00ed v\u011btvi, byly servery nap\u00e1jeny pouze p\u0159es jednu nap\u00e1jec\u00ed v\u011btev, co\u017e je velmi riskantn\u00ed, tak jsme se po dohod\u011b s dodavatelem UPS domluvili na tom, \u017ee po nabit\u00ed bateri\u00ed funk\u010dn\u00ed UPS, p\u0159epneme po\u0161kozenou UPS do re\u017eimu bypass a t\u00edm zajist\u00edme server\u016fm nap\u00e1jen\u00ed i z druh\u00e9 v\u011btve. Tato operace byla napl\u00e1nov\u00e1na na 8 hodin r\u00e1no a m\u011blo se jednat o rutinn\u00ed operaci bez jak\u00e9hokoliv v\u00fdpadku.<\/p>\n\n\n\n
R\u00e1no p\u0159i\u0161el technik na opravu UPS (ta byla p\u0159epnuta na bypass). V\u0161e se zd\u00e1lo b\u00fdt vy\u0159e\u0161eno, ale oprava, a\u010dkoliv se zd\u00e1la v po\u0159\u00e1dku (dle ve\u0161ker\u00fdch dostupn\u00fdch m\u011b\u0159en\u00ed), neprob\u011bhla dob\u0159e, resp. neodhalila vadnou sou\u010d\u00e1stku. Po p\u0159epnut\u00ed UPS do re\u017eimu bypass do\u0161lo ke zkratu a t\u00edm p\u0159eru\u0161en\u00ed nap\u00e1jen\u00ed. Bohu\u017eel baterie, kter\u00e9 fungovaly jako z\u00e1lo\u017en\u00ed zdroj v p\u00e1tek nave\u010der, nebyly je\u0161t\u011b zcela nabit\u00e9, tak\u017ee nap\u00e1jely servery 20 minut. Ne\u017e stihl technik p\u0159emontovat nov\u00fd hlavn\u00ed jisti\u010d, nastal druh\u00fd v\u00fdpadek a to na cel\u00fdch 15 minut. Po tomto v\u00fdpadku na\u0161i technici v\u0161echny slu\u017eby obnovili ve velmi kr\u00e1tk\u00e9 dob\u011b.<\/p>\n\n\n\n
Cel\u00fd zbytek dne jsme pak \u0159e\u0161ili, jak podobn\u00e9 situaci jednou pro v\u017edy p\u0159edej\u00edt. Gener\u00e1tor byl je\u0161t\u011b v sobotu opraven a v\u011b\u0159\u00edme, \u017ee podobn\u00e1 souhra tolika nepravd\u011bpodobn\u00fdch ud\u00e1lost\u00ed najednou, se ji\u017e nebude opakovat. A i kdyby nastala, jsme na ni l\u00e9pe p\u0159ipraveni. V pond\u011bl\u00ed tak\u00e9 objedn\u00e1v\u00e1me \u00fapln\u011b novou UPS. O dal\u0161\u00edch vylep\u0161en\u00edch, kter\u00e1 maj\u00ed p\u0159edej\u00edt podobn\u00fdm v\u00fdpadk\u016fm, v\u00e1s budeme pr\u016fb\u011b\u017en\u011b informovat.<\/p>\n\n\n\n
V\u011b\u0159te, \u017ee nikdo zde nezah\u00e1lel a situaci jsme brali velmi v\u00e1\u017en\u011b. Do pr\u00e1ce p\u0159i\u0161li i kolegov\u00e9, kte\u0159\u00ed nem\u011bli zrovna sm\u011bnu a sna\u017eili se pomoci. Jakmile bylo mo\u017en\u00e9 zvedat telefony a odpov\u00eddat na Va\u0161e dotazy, bylo zde k dispozici 10 administr\u00e1tor\u016f, kte\u0159\u00ed trp\u011bliv\u011b odpov\u00eddali a vysv\u011btlovali.<\/p>\n\n\n\n
Jsme si v\u011bdomi toho, \u017ee ve chv\u00edli, kdy u n\u00e1s hostuj\u00ed des\u00edtky tis\u00edc klient\u016f mus\u00ed v\u0161e fungovat na 100% a i ochrann\u00e9 mechanismy, kter\u00e9 \u0159e\u0161\u00ed v\u00fdpadky elektriky, \u00fatoky atd., mus\u00ed m\u00edt je\u0161t\u011b z\u00e1lo\u017en\u00ed \u0159e\u0161en\u00ed a dal\u0161\u00ed mo\u017enosti, jak v\u0161e zvl\u00e1dnout tak, aby klient nemusel poci\u0165ovat nic jin\u00e9ho ne\u017e spokojenost s na\u0161imi slu\u017ebami.<\/p>\n\n\n\n
V\u0161em z\u00e1kazn\u00edk\u016fm, kter\u00fdch se probl\u00e9my dotkly, poskytneme automaticky kompenzaci v podob\u011b slu\u017eeb zdarma (budeme \u0159e\u0161it v pr\u016fb\u011bhu p\u0159\u00ed\u0161t\u00edho m\u011bs\u00edce). Z\u00e1kazn\u00edci, kte\u0159\u00ed maj\u00ed dle smluvn\u00edch podm\u00ednek n\u00e1rok na vy\u0161\u0161\u00ed kompenzace, budou \u0159e\u0161eni individu\u00e1ln\u011b.<\/p>\n\n\n\n
Z v\u00fd\u0161e uveden\u00fdch probl\u00e9m\u016f se pou\u010d\u00edme a je\u0161t\u011b jednou se omlouv\u00e1me v\u0161em na\u0161im klient\u016fm. D\u011bkujeme zam\u011bstnanc\u016fm, dodavatel\u016fm a spolupracuj\u00edc\u00edm firm\u00e1m za \u0159e\u0161en\u00ed cel\u00e9 situace. A v\u011b\u0159\u00edme, \u017ee je velmi nepravd\u011bpodobn\u00e9, aby se podobn\u00e1 shoda n\u00e1hod opakovala, proto\u017ee do\u0161lo k po\u0161kozen\u00ed nejen prim\u00e1rn\u00edch, ale i z\u00e1lo\u017en\u00edch zdroj\u016f a t\u00edm k velk\u00fdm komplikac\u00edm.<\/p>\n\n\n\n
Aktu\u00e1ln\u011b mohou m\u00edt st\u00e1le probl\u00e9m z\u00e1kazn\u00edci, kter\u00fdm se VPS zastavila na v\u00fdb\u011bru p\u0159ed zaveden\u00edm OS (Grub, n\u00e1stroj na opravu po p\u00e1du, …). \u0158e\u0161en\u00ed je KVM. V nejhor\u0161\u00edm p\u0159\u00edpad\u011b provedeme obnoven\u00ed ze z\u00e1lohy zdarma. D\u00e1le mohou b\u00fdt po\u0161kozeny n\u011bkter\u00e9 tabulky u datab\u00e1z\u00ed webhostingu a je nutn\u00e9 n\u00e1s kontaktovat, abychom s jejich opravou pomohli. Odhadujeme, \u017ee tyto probl\u00e9my se dot\u00fdkaj\u00ed n\u011bkolika des\u00edtek z\u00e1kazn\u00edk\u016f a bohu\u017eel nen\u00ed v na\u0161ich sil\u00e1ch zjistit, koho konkr\u00e9tn\u011b se to t\u00fdk\u00e1 a je tedy nutn\u00e9, aby n\u00e1s doty\u010dn\u00ed z\u00e1kazn\u00edci sami oslovili.<\/p>\n\n\n\n
Na z\u00e1v\u011br se je\u0161t\u011b jednou v\u0161em z\u00e1kazn\u00edk\u016fm omlouv\u00e1me za probl\u00e9my a z\u00e1rove\u0148 za omezenou mo\u017enost komunikace v okam\u017eiku v\u00fdpadku. Ji\u017e jsme u n\u00e1s vy\u0159e\u0161ili n\u00e1hradn\u00ed zp\u016fsoby komunikace v p\u0159\u00edpad\u011b podobn\u00e9 krizov\u00e9 situace. Od p\u00e1te\u010dn\u00edho odpoledne jsme se v\u011bnovali na 100% \u0159e\u0161en\u00ed cel\u00e9 situace a to a\u017e do dne\u0161n\u00ed 11 hodiny, kdy jsou ji\u017e v\u0161echna po\u0161kozen\u00e1 za\u0159\u00edzen\u00ed pln\u011b funk\u010dn\u00ed (s v\u00fdjimkou jedn\u00e9 UPS, kter\u00e1 bude nahrazena novou).<\/p>\n","protected":false},"excerpt":{"rendered":"
Informa\u010dn\u00ed zpr\u00e1va o technick\u00fdch probl\u00e9mech v na\u0161em datacentru v odpoledn\u00edch hodin\u00e1ch dne 23 a v rann\u00edch hodin\u00e1ch dne 24. kv\u011btna 2014.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[101],"tags":[],"class_list":["post-54998","post","type-post","status-publish","format-standard","hentry","category-udalosti"],"_links":{"self":[{"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/posts\/54998","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/comments?post=54998"}],"version-history":[{"count":1,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/posts\/54998\/revisions"}],"predecessor-version":[{"id":55004,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/posts\/54998\/revisions\/55004"}],"wp:attachment":[{"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/media?parent=54998"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/categories?post=54998"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.wedos.com\/cs\/wp-json\/wp\/v2\/tags?post=54998"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}