Created
February 23, 2024 12:36
-
-
Save alfondotnet/0458178917bd1a5b8fc1c46c0b91dcc6 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta content="width=device-width, initial-scale=1" name="viewport"/><link href="/static/favicons/favicon@57x57.png" rel="apple-touch-icon" sizes="57x57"/><link href="/static/favicons/favicon@60x60.png" rel="apple-touch-icon" sizes="60x60"/><link href="/static/favicons/favicon@72x72.png" rel="apple-touch-icon" sizes="72x72"/><link href="/static/favicons/favicon@76x76.png" rel="apple-touch-icon" sizes="76x76"/><link href="/static/favicons/favicon@114x114.png" rel="apple-touch-icon" sizes="114x114"/><link href="/static/favicons/favicon@120x120.png" rel="apple-touch-icon" sizes="120x120"/><link href="/static/favicons/favicon@144x144.png" rel="apple-touch-icon" sizes="144x144"/><link href="/static/favicons/favicon@152x152.png" rel="apple-touch-icon" sizes="152x152"/><link href="/static/favicons/favicon@180x180.png" rel="apple-touch-icon" sizes="180x180"/><link href="/static/favicons/favicon.ico" rel="icon shortcut" type="image/x-icon"/><meta content="summary_large_image" name="twitter:card"/><meta content="website" property="og:type"/><title>Incident report for February 21st, 2024 · Resend</title><meta content="Incident report for February 21st, 2024 · Resend" property="og:title"/><meta content="Detailed postmortem of the outage on February 21st." name="description"/><meta content="Detailed postmortem of the outage on February 21st." property="og:description"/><meta content="https://resend.com/static/posts/incident-report-for-february-21-2024.jpg" name="twitter:image"/><meta content="https://resend.com/static/posts/incident-report-for-february-21-2024.jpg" name="og:image"/><meta content="https://resend.com/blog/incident-report-for-february-21-2024" property="og:url"/><link rel="preload" as="image" imageSrcSet="/_next/image?url=%2Fstatic%2Favatars%2Fbu-kinoshita.jpg&w=32&q=75 1x, /_next/image?url=%2Fstatic%2Favatars%2Fbu-kinoshita.jpg&w=64&q=75 2x" fetchpriority="high"/><link rel="preload" as="image" imageSrcSet="/_next/image?url=%2Fstatic%2Fposts%2Fincident-report-for-february-21-2024.jpg&w=1200&q=100 1x, /_next/image?url=%2Fstatic%2Fposts%2Fincident-report-for-february-21-2024.jpg&w=3840&q=100 2x" fetchpriority="high"/><meta name="next-head-count" content="23"/><link rel="preload" href="/_next/static/media/c9a5bc6a7c948fb0-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/media/988ffee4f1b4b3a7-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/css/6b7a5a9074fce82d.css" as="style" crossorigin=""/><link rel="stylesheet" href="/_next/static/css/6b7a5a9074fce82d.css" crossorigin="" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" crossorigin="" nomodule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-8250682d43598da5.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/framework-32cec10819b332f0.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/main-1dea9989d67ba28e.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/pages/_app-f206cd27dd3c8d77.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/41664-00ea70f7bdba1450.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/51327-2cc375ff2d748220.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/25675-77d18f6c85f6ab17.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/91105-1629ea7fa8645f21.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/53176-e5d94e07a41a6ed9.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/20731-d596d036beb3879c.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/21871-d1947e8582869337.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/7287-e4779d6426c5dd49.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/77472-8d862f97e4c0fa53.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/68161-13415c3134f3bcc0.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/68374-93ba55e3f57d83e0.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/33554-c532eb107dc5361c.js" defer="" crossorigin=""></script><script src="/_next/static/chunks/pages/blog/%5Bslug%5D-1ef5be24d8af14c2.js" defer="" crossorigin=""></script><script src="/_next/static/XSe6_RzH2FbEfcRaaL7by/_buildManifest.js" defer="" crossorigin=""></script><script src="/_next/static/XSe6_RzH2FbEfcRaaL7by/_ssgManifest.js" defer="" crossorigin=""></script></head><body class="bg-root relative font-sans text-slate-12 antialiased selection:bg-mint-4 selection:text-mint-11"><div id="__next"><script>!function(){var d=document.documentElement,c=d.classList;c.remove('light','dark');d.style.colorScheme = 'dark';c.add('dark')}()</script><div class="__variable_469f07 __variable_bf2c8d font-sans"><div class="bg-black"><header class="sticky top-0 border-b border-transparent backdrop-blur-sm transition duration-200 ease-in-out" style="z-index:39"><div class="md:max-w-7xl mx-auto w-full max-w-5xl px-6"><div class="bg-root absolute left-0 top-0 z-20 flex w-full flex-col items-center md:hidden"><div class="flex w-full items-center px-6 py-4"><div class="flex-auto"><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7" href="/home"><svg fill="none" viewBox="0 0 65 16" width="60" xmlns="http://www.w3.org/2000/svg"><path d="M0.820068 15V1.00001H7.02007C7.88674 1.00001 8.6734 1.20001 9.38007 1.60001C10.0867 1.98668 10.6401 2.51334 11.0401 3.18001C11.4534 3.84668 11.6601 4.60668 11.6601 5.46001C11.6601 6.30001 11.4534 7.06668 11.0401 7.76001C10.6401 8.44001 10.0867 8.98001 9.38007 9.38001C8.6734 9.78001 7.88674 9.98001 7.02007 9.98001H3.72007V15H0.820068ZM8.76007 15L5.20007 8.68001L8.28007 8.18001L12.2401 15.02L8.76007 15ZM3.72007 7.54001H6.88007C7.24007 7.54001 7.5534 7.46001 7.82007 7.30001C8.10007 7.12668 8.3134 6.89334 8.46007 6.60001C8.60673 6.29335 8.68007 5.95335 8.68007 5.58001C8.68007 5.18001 8.5934 4.83335 8.42007 4.54001C8.24674 4.24668 7.9934 4.02001 7.66007 3.86001C7.32674 3.68668 6.94007 3.60001 6.50007 3.60001H3.72007V7.54001Z" fill="currentColor"></path><path d="M18.0534 15.2C16.9067 15.2 15.9 14.9667 15.0333 14.5C14.18 14.0333 13.5134 13.3933 13.0333 12.58C12.5667 11.7667 12.3333 10.8333 12.3333 9.78001C12.3333 8.95335 12.4667 8.20001 12.7333 7.52001C13 6.84001 13.3733 6.25335 13.8533 5.76001C14.3333 5.25335 14.9 4.86668 15.5534 4.60001C16.22 4.32001 16.94 4.18001 17.7134 4.18001C18.4334 4.18001 19.1 4.31335 19.7134 4.58001C20.3267 4.84668 20.8534 5.22001 21.2934 5.70001C21.7467 6.16668 22.0934 6.72668 22.3334 7.38001C22.5734 8.02001 22.68 8.71335 22.6534 9.46001L22.6334 10.34H14.1334L13.6733 8.60001H20.2934L19.9734 8.96001V8.52001C19.9467 8.16001 19.8267 7.84001 19.6133 7.56001C19.4134 7.26668 19.1534 7.04001 18.8334 6.88001C18.5134 6.70668 18.1533 6.62001 17.7533 6.62001C17.1667 6.62001 16.6667 6.73335 16.2533 6.96001C15.8533 7.18668 15.5467 7.52001 15.3333 7.96001C15.12 8.40001 15.0133 8.93335 15.0133 9.56001C15.0133 10.2 15.1467 10.7533 15.4134 11.22C15.6934 11.6867 16.08 12.0533 16.5734 12.32C17.08 12.5733 17.6733 12.7 18.3533 12.7C18.82 12.7 19.2467 12.6267 19.6334 12.48C20.02 12.3333 20.4333 12.08 20.8734 11.72L22.2334 13.62C21.8467 13.9667 21.42 14.26 20.9534 14.5C20.4867 14.7267 20.0067 14.9 19.5133 15.02C19.02 15.14 18.5334 15.2 18.0534 15.2Z" fill="currentColor"></path><path d="M27.3121 15.2C26.3254 15.2 25.4454 15.04 24.6721 14.72C23.9121 14.3867 23.2988 13.9333 22.8321 13.36L24.6121 11.84C25.0121 12.28 25.4654 12.6 25.9721 12.8C26.4788 12.9867 26.9854 13.08 27.4921 13.08C27.6921 13.08 27.8721 13.06 28.0321 13.02C28.2054 12.9667 28.3521 12.9 28.4721 12.82C28.5921 12.7267 28.6788 12.62 28.7321 12.5C28.7988 12.3667 28.8321 12.2267 28.8321 12.08C28.8321 11.7867 28.7121 11.56 28.4721 11.4C28.3388 11.32 28.1321 11.2333 27.8521 11.14C27.5721 11.0333 27.2121 10.92 26.7721 10.8C26.0921 10.6267 25.5121 10.4267 25.0321 10.2C24.5654 9.96001 24.1921 9.69335 23.9121 9.40001C23.6721 9.12001 23.4854 8.82001 23.3521 8.50001C23.2321 8.16668 23.1721 7.80001 23.1721 7.40001C23.1721 6.92001 23.2788 6.48668 23.4921 6.10001C23.7054 5.70001 23.9988 5.36001 24.3721 5.08001C24.7588 4.80001 25.1988 4.58668 25.6921 4.44001C26.1854 4.28001 26.7054 4.20001 27.2521 4.20001C27.7988 4.20001 28.3321 4.26668 28.8521 4.40001C29.3721 4.53335 29.8521 4.72668 30.2921 4.98001C30.7454 5.22001 31.1388 5.50668 31.4721 5.84001L29.9521 7.52001C29.7121 7.29334 29.4388 7.08668 29.1321 6.90001C28.8388 6.71335 28.5321 6.56668 28.2121 6.46001C27.8921 6.35335 27.6054 6.30001 27.3521 6.30001C27.1254 6.30001 26.9188 6.32001 26.7321 6.36001C26.5588 6.40001 26.4121 6.46668 26.2921 6.56001C26.1721 6.64001 26.0788 6.74001 26.0121 6.86001C25.9588 6.98001 25.9321 7.11334 25.9321 7.26001C25.9321 7.40668 25.9654 7.54668 26.0321 7.68001C26.1121 7.81335 26.2188 7.92668 26.3521 8.02001C26.4988 8.10001 26.7121 8.19335 26.9921 8.30001C27.2854 8.40668 27.6788 8.52668 28.1721 8.66001C28.8121 8.83335 29.3521 9.02668 29.7921 9.24001C30.2454 9.45335 30.6054 9.70001 30.8721 9.98001C31.0988 10.22 31.2654 10.4933 31.3721 10.8C31.4788 11.1067 31.5321 11.4467 31.5321 11.82C31.5321 12.4733 31.3454 13.0533 30.9721 13.56C30.6121 14.0667 30.1121 14.4667 29.4721 14.76C28.8321 15.0533 28.1121 15.2 27.3121 15.2Z" fill="currentColor"></path><path d="M37.5768 15.2C36.4301 15.2 35.4235 14.9667 34.5568 14.5C33.7035 14.0333 33.0368 13.3933 32.5568 12.58C32.0901 11.7667 31.8568 10.8333 31.8568 9.78001C31.8568 8.95335 31.9901 8.20001 32.2568 7.52001C32.5235 6.84001 32.8968 6.25335 33.3768 5.76001C33.8568 5.25335 34.4235 4.86668 35.0768 4.60001C35.7435 4.32001 36.4635 4.18001 37.2368 4.18001C37.9568 4.18001 38.6235 4.31335 39.2368 4.58001C39.8501 4.84668 40.3768 5.22001 40.8168 5.70001C41.2701 6.16668 41.6168 6.72668 41.8568 7.38001C42.0968 8.02001 42.2035 8.71335 42.1768 9.46001L42.1568 10.34H33.6568L33.1968 8.60001H39.8168L39.4968 8.96001V8.52001C39.4701 8.16001 39.3501 7.84001 39.1368 7.56001C38.9368 7.26668 38.6768 7.04001 38.3568 6.88001C38.0368 6.70668 37.6768 6.62001 37.2768 6.62001C36.6901 6.62001 36.1901 6.73335 35.7768 6.96001C35.3768 7.18668 35.0701 7.52001 34.8568 7.96001C34.6435 8.40001 34.5368 8.93335 34.5368 9.56001C34.5368 10.2 34.6701 10.7533 34.9368 11.22C35.2168 11.6867 35.6035 12.0533 36.0968 12.32C36.6035 12.5733 37.1968 12.7 37.8768 12.7C38.3435 12.7 38.7701 12.6267 39.1568 12.48C39.5435 12.3333 39.9568 12.08 40.3968 11.72L41.7568 13.62C41.3701 13.9667 40.9435 14.26 40.4768 14.5C40.0101 14.7267 39.5301 14.9 39.0368 15.02C38.5435 15.14 38.0568 15.2 37.5768 15.2Z" fill="currentColor"></path><path d="M43.2755 15V4.42001H45.9955L46.0755 6.58001L45.5155 6.82001C45.6622 6.34001 45.9222 5.90668 46.2955 5.52001C46.6822 5.12001 47.1422 4.80001 47.6755 4.56001C48.2089 4.32001 48.7689 4.20001 49.3555 4.20001C50.1555 4.20001 50.8222 4.36001 51.3555 4.68001C51.9022 5.00001 52.3089 5.48668 52.5755 6.14001C52.8555 6.78001 52.9955 7.57335 52.9955 8.52001V15H50.1555V8.74001C50.1555 8.26001 50.0889 7.86001 49.9555 7.54001C49.8222 7.22001 49.6155 6.98668 49.3355 6.84001C49.0689 6.69334 48.7355 6.62668 48.3355 6.64001C48.0155 6.64001 47.7155 6.69335 47.4355 6.80001C47.1689 6.89334 46.9355 7.03335 46.7355 7.22001C46.5489 7.40668 46.3955 7.62001 46.2755 7.86001C46.1689 8.10001 46.1155 8.36001 46.1155 8.64001V15H44.7155C44.4089 15 44.1355 15 43.8955 15C43.6555 15 43.4489 15 43.2755 15Z" fill="currentColor"></path><path d="M58.8569 15.2C57.9236 15.2 57.0903 14.9667 56.3569 14.5C55.6369 14.02 55.0636 13.3733 54.6369 12.56C54.2236 11.7333 54.0169 10.78 54.0169 9.70001C54.0169 8.64668 54.2236 7.70668 54.6369 6.88001C55.0636 6.04001 55.6369 5.38668 56.3569 4.92001C57.0903 4.44001 57.9236 4.20001 58.8569 4.20001C59.3503 4.20001 59.8236 4.28001 60.2769 4.44001C60.7436 4.58668 61.1569 4.79335 61.5169 5.06001C61.8903 5.32668 62.1903 5.62668 62.4169 5.96001C62.6436 6.28001 62.7703 6.61335 62.7969 6.96001L62.0769 7.10001V0.200012H64.9369V15H62.2369L62.1169 12.56L62.6769 12.62C62.6503 12.9533 62.5303 13.2733 62.3169 13.58C62.1036 13.8867 61.8169 14.1667 61.4569 14.42C61.1103 14.66 60.7103 14.8533 60.2569 15C59.8169 15.1333 59.3503 15.2 58.8569 15.2ZM59.4969 12.84C60.0303 12.84 60.4969 12.7067 60.8969 12.44C61.2969 12.1733 61.6103 11.8067 61.8369 11.34C62.0636 10.8733 62.1769 10.3267 62.1769 9.70001C62.1769 9.08668 62.0636 8.54668 61.8369 8.08001C61.6103 7.61335 61.2969 7.24668 60.8969 6.98001C60.4969 6.71335 60.0303 6.58001 59.4969 6.58001C58.9636 6.58001 58.4969 6.71335 58.0969 6.98001C57.7103 7.24668 57.4036 7.61335 57.1769 8.08001C56.9636 8.54668 56.8569 9.08668 56.8569 9.70001C56.8569 10.3267 56.9636 10.8733 57.1769 11.34C57.4036 11.8067 57.7103 12.1733 58.0969 12.44C58.4969 12.7067 58.9636 12.84 59.4969 12.84Z" fill="currentColor"></path></svg></a></div><div class="flex flex-auto justify-end"><button aria-controls="mobile-menu" aria-expanded="false" aria-label="menu" class="inline-flex items-center justify-center rounded-md p-1 text-slate-11 transition ease-in-out hover:bg-slate-5 hover:text-slate-12 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-slate-6" type="button"><span class="sr-only">Open main menu</span><svg aria-hidden="true" class="block h-8 w-8" fill="none" stroke="currentColor" stroke-width="1.5" viewBox="0 0 24 24"><path d="M3.75 6.75h16.5M3.75 12h16.5m-16.5 5.25h16.5" stroke-linecap="round" stroke-linejoin="round"></path></svg></button></div></div><div class="flex w-full px-6 py-4 md:hidden hidden" id="mobile-menu"><div class="flex w-full flex-col"><a class="text-base h-11 pl-4 pr-4 rounded-md gap-2 font-semibold dark:bg-slate-3 border-slate-6 dark:text-slate-11 bg-slate-2 text-slate-12 hover:bg-slate-4 focus-visible:ring-2 focus-visible:ring-slate-7 focus-visible:outline-none focus-visible:bg-slate-4 disabled:hover:bg-slate-4 inline-flex items-center border justify-center select-none disabled:cursor-not-allowed disabled:opacity-70 transition ease-in-out duration-200 mb-4 w-full" href="/login">Sign In</a><a class="text-base h-11 pl-4 pr-4 rounded-md gap-2 font-semibold bg-black dark:bg-white text-white dark:text-black border-slate-6 hover:bg-black/90 dark:hover:bg-white/90 focus-visible:ring-2 dark:focus-visible:ring-white/40 focus-visible:ring-black/40 focus-visible:outline-none dark:focus-visible:bg-white/90 focus-visible:bg-black/90 disabled:hover:bg-black dark:disabled:hover:bg-white inline-flex items-center border justify-center select-none disabled:cursor-not-allowed disabled:opacity-70 transition ease-in-out duration-200 mb-4 w-full" href="/signup">Get Started</a><a class="text-md block w-full border-b border-slate-6 py-4 font-semibold text-slate-11 transition duration-200 ease-in-out last:border-none hover:text-slate-12" href="/about">About</a><a class="text-md block w-full border-b border-slate-6 py-4 font-semibold text-slate-11 transition duration-200 ease-in-out last:border-none hover:text-slate-12" href="/blog">Blog</a><a class="text-md block w-full border-b border-slate-6 py-4 font-semibold text-slate-11 transition duration-200 ease-in-out last:border-none hover:text-slate-12" href="/customers">Customers</a><a class="text-md block w-full border-b border-slate-6 py-4 font-semibold text-slate-11 transition duration-200 ease-in-out last:border-none hover:text-slate-12" href="/pricing">Pricing</a><a class="text-md block w-full border-b border-slate-6 py-4 font-semibold text-slate-11 transition duration-200 ease-in-out last:border-none hover:text-slate-12" href="/enterprise">Enterprise</a><a class="text-md block w-full border-b border-slate-6 py-4 font-semibold text-slate-11 transition duration-200 ease-in-out last:border-none hover:text-slate-12" href="/changelog">Changelog</a><a class="text-md block w-full border-b border-slate-6 py-4 font-semibold text-slate-11 transition duration-200 ease-in-out last:border-none hover:text-slate-12" href="/docs">Docs</a><a class="text-md block w-full border-b border-slate-6 py-4 font-semibold text-slate-11 transition duration-200 ease-in-out last:border-none hover:text-slate-12" href="/contact">Contact</a></div></div></div><div class="mx-auto hidden h-[58px] w-full items-center justify-between transition duration-500 ease-in-out md:flex"><div class="flex lg:w-[225px]"><span data-state="closed" class="" style="-webkit-touch-callout:none"><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 py-1" aria-label="Resend" href="/home"><svg fill="none" viewBox="0 0 65 16" width="70" xmlns="http://www.w3.org/2000/svg"><path d="M0.820068 15V1.00001H7.02007C7.88674 1.00001 8.6734 1.20001 9.38007 1.60001C10.0867 1.98668 10.6401 2.51334 11.0401 3.18001C11.4534 3.84668 11.6601 4.60668 11.6601 5.46001C11.6601 6.30001 11.4534 7.06668 11.0401 7.76001C10.6401 8.44001 10.0867 8.98001 9.38007 9.38001C8.6734 9.78001 7.88674 9.98001 7.02007 9.98001H3.72007V15H0.820068ZM8.76007 15L5.20007 8.68001L8.28007 8.18001L12.2401 15.02L8.76007 15ZM3.72007 7.54001H6.88007C7.24007 7.54001 7.5534 7.46001 7.82007 7.30001C8.10007 7.12668 8.3134 6.89334 8.46007 6.60001C8.60673 6.29335 8.68007 5.95335 8.68007 5.58001C8.68007 5.18001 8.5934 4.83335 8.42007 4.54001C8.24674 4.24668 7.9934 4.02001 7.66007 3.86001C7.32674 3.68668 6.94007 3.60001 6.50007 3.60001H3.72007V7.54001Z" fill="currentColor"></path><path d="M18.0534 15.2C16.9067 15.2 15.9 14.9667 15.0333 14.5C14.18 14.0333 13.5134 13.3933 13.0333 12.58C12.5667 11.7667 12.3333 10.8333 12.3333 9.78001C12.3333 8.95335 12.4667 8.20001 12.7333 7.52001C13 6.84001 13.3733 6.25335 13.8533 5.76001C14.3333 5.25335 14.9 4.86668 15.5534 4.60001C16.22 4.32001 16.94 4.18001 17.7134 4.18001C18.4334 4.18001 19.1 4.31335 19.7134 4.58001C20.3267 4.84668 20.8534 5.22001 21.2934 5.70001C21.7467 6.16668 22.0934 6.72668 22.3334 7.38001C22.5734 8.02001 22.68 8.71335 22.6534 9.46001L22.6334 10.34H14.1334L13.6733 8.60001H20.2934L19.9734 8.96001V8.52001C19.9467 8.16001 19.8267 7.84001 19.6133 7.56001C19.4134 7.26668 19.1534 7.04001 18.8334 6.88001C18.5134 6.70668 18.1533 6.62001 17.7533 6.62001C17.1667 6.62001 16.6667 6.73335 16.2533 6.96001C15.8533 7.18668 15.5467 7.52001 15.3333 7.96001C15.12 8.40001 15.0133 8.93335 15.0133 9.56001C15.0133 10.2 15.1467 10.7533 15.4134 11.22C15.6934 11.6867 16.08 12.0533 16.5734 12.32C17.08 12.5733 17.6733 12.7 18.3533 12.7C18.82 12.7 19.2467 12.6267 19.6334 12.48C20.02 12.3333 20.4333 12.08 20.8734 11.72L22.2334 13.62C21.8467 13.9667 21.42 14.26 20.9534 14.5C20.4867 14.7267 20.0067 14.9 19.5133 15.02C19.02 15.14 18.5334 15.2 18.0534 15.2Z" fill="currentColor"></path><path d="M27.3121 15.2C26.3254 15.2 25.4454 15.04 24.6721 14.72C23.9121 14.3867 23.2988 13.9333 22.8321 13.36L24.6121 11.84C25.0121 12.28 25.4654 12.6 25.9721 12.8C26.4788 12.9867 26.9854 13.08 27.4921 13.08C27.6921 13.08 27.8721 13.06 28.0321 13.02C28.2054 12.9667 28.3521 12.9 28.4721 12.82C28.5921 12.7267 28.6788 12.62 28.7321 12.5C28.7988 12.3667 28.8321 12.2267 28.8321 12.08C28.8321 11.7867 28.7121 11.56 28.4721 11.4C28.3388 11.32 28.1321 11.2333 27.8521 11.14C27.5721 11.0333 27.2121 10.92 26.7721 10.8C26.0921 10.6267 25.5121 10.4267 25.0321 10.2C24.5654 9.96001 24.1921 9.69335 23.9121 9.40001C23.6721 9.12001 23.4854 8.82001 23.3521 8.50001C23.2321 8.16668 23.1721 7.80001 23.1721 7.40001C23.1721 6.92001 23.2788 6.48668 23.4921 6.10001C23.7054 5.70001 23.9988 5.36001 24.3721 5.08001C24.7588 4.80001 25.1988 4.58668 25.6921 4.44001C26.1854 4.28001 26.7054 4.20001 27.2521 4.20001C27.7988 4.20001 28.3321 4.26668 28.8521 4.40001C29.3721 4.53335 29.8521 4.72668 30.2921 4.98001C30.7454 5.22001 31.1388 5.50668 31.4721 5.84001L29.9521 7.52001C29.7121 7.29334 29.4388 7.08668 29.1321 6.90001C28.8388 6.71335 28.5321 6.56668 28.2121 6.46001C27.8921 6.35335 27.6054 6.30001 27.3521 6.30001C27.1254 6.30001 26.9188 6.32001 26.7321 6.36001C26.5588 6.40001 26.4121 6.46668 26.2921 6.56001C26.1721 6.64001 26.0788 6.74001 26.0121 6.86001C25.9588 6.98001 25.9321 7.11334 25.9321 7.26001C25.9321 7.40668 25.9654 7.54668 26.0321 7.68001C26.1121 7.81335 26.2188 7.92668 26.3521 8.02001C26.4988 8.10001 26.7121 8.19335 26.9921 8.30001C27.2854 8.40668 27.6788 8.52668 28.1721 8.66001C28.8121 8.83335 29.3521 9.02668 29.7921 9.24001C30.2454 9.45335 30.6054 9.70001 30.8721 9.98001C31.0988 10.22 31.2654 10.4933 31.3721 10.8C31.4788 11.1067 31.5321 11.4467 31.5321 11.82C31.5321 12.4733 31.3454 13.0533 30.9721 13.56C30.6121 14.0667 30.1121 14.4667 29.4721 14.76C28.8321 15.0533 28.1121 15.2 27.3121 15.2Z" fill="currentColor"></path><path d="M37.5768 15.2C36.4301 15.2 35.4235 14.9667 34.5568 14.5C33.7035 14.0333 33.0368 13.3933 32.5568 12.58C32.0901 11.7667 31.8568 10.8333 31.8568 9.78001C31.8568 8.95335 31.9901 8.20001 32.2568 7.52001C32.5235 6.84001 32.8968 6.25335 33.3768 5.76001C33.8568 5.25335 34.4235 4.86668 35.0768 4.60001C35.7435 4.32001 36.4635 4.18001 37.2368 4.18001C37.9568 4.18001 38.6235 4.31335 39.2368 4.58001C39.8501 4.84668 40.3768 5.22001 40.8168 5.70001C41.2701 6.16668 41.6168 6.72668 41.8568 7.38001C42.0968 8.02001 42.2035 8.71335 42.1768 9.46001L42.1568 10.34H33.6568L33.1968 8.60001H39.8168L39.4968 8.96001V8.52001C39.4701 8.16001 39.3501 7.84001 39.1368 7.56001C38.9368 7.26668 38.6768 7.04001 38.3568 6.88001C38.0368 6.70668 37.6768 6.62001 37.2768 6.62001C36.6901 6.62001 36.1901 6.73335 35.7768 6.96001C35.3768 7.18668 35.0701 7.52001 34.8568 7.96001C34.6435 8.40001 34.5368 8.93335 34.5368 9.56001C34.5368 10.2 34.6701 10.7533 34.9368 11.22C35.2168 11.6867 35.6035 12.0533 36.0968 12.32C36.6035 12.5733 37.1968 12.7 37.8768 12.7C38.3435 12.7 38.7701 12.6267 39.1568 12.48C39.5435 12.3333 39.9568 12.08 40.3968 11.72L41.7568 13.62C41.3701 13.9667 40.9435 14.26 40.4768 14.5C40.0101 14.7267 39.5301 14.9 39.0368 15.02C38.5435 15.14 38.0568 15.2 37.5768 15.2Z" fill="currentColor"></path><path d="M43.2755 15V4.42001H45.9955L46.0755 6.58001L45.5155 6.82001C45.6622 6.34001 45.9222 5.90668 46.2955 5.52001C46.6822 5.12001 47.1422 4.80001 47.6755 4.56001C48.2089 4.32001 48.7689 4.20001 49.3555 4.20001C50.1555 4.20001 50.8222 4.36001 51.3555 4.68001C51.9022 5.00001 52.3089 5.48668 52.5755 6.14001C52.8555 6.78001 52.9955 7.57335 52.9955 8.52001V15H50.1555V8.74001C50.1555 8.26001 50.0889 7.86001 49.9555 7.54001C49.8222 7.22001 49.6155 6.98668 49.3355 6.84001C49.0689 6.69334 48.7355 6.62668 48.3355 6.64001C48.0155 6.64001 47.7155 6.69335 47.4355 6.80001C47.1689 6.89334 46.9355 7.03335 46.7355 7.22001C46.5489 7.40668 46.3955 7.62001 46.2755 7.86001C46.1689 8.10001 46.1155 8.36001 46.1155 8.64001V15H44.7155C44.4089 15 44.1355 15 43.8955 15C43.6555 15 43.4489 15 43.2755 15Z" fill="currentColor"></path><path d="M58.8569 15.2C57.9236 15.2 57.0903 14.9667 56.3569 14.5C55.6369 14.02 55.0636 13.3733 54.6369 12.56C54.2236 11.7333 54.0169 10.78 54.0169 9.70001C54.0169 8.64668 54.2236 7.70668 54.6369 6.88001C55.0636 6.04001 55.6369 5.38668 56.3569 4.92001C57.0903 4.44001 57.9236 4.20001 58.8569 4.20001C59.3503 4.20001 59.8236 4.28001 60.2769 4.44001C60.7436 4.58668 61.1569 4.79335 61.5169 5.06001C61.8903 5.32668 62.1903 5.62668 62.4169 5.96001C62.6436 6.28001 62.7703 6.61335 62.7969 6.96001L62.0769 7.10001V0.200012H64.9369V15H62.2369L62.1169 12.56L62.6769 12.62C62.6503 12.9533 62.5303 13.2733 62.3169 13.58C62.1036 13.8867 61.8169 14.1667 61.4569 14.42C61.1103 14.66 60.7103 14.8533 60.2569 15C59.8169 15.1333 59.3503 15.2 58.8569 15.2ZM59.4969 12.84C60.0303 12.84 60.4969 12.7067 60.8969 12.44C61.2969 12.1733 61.6103 11.8067 61.8369 11.34C62.0636 10.8733 62.1769 10.3267 62.1769 9.70001C62.1769 9.08668 62.0636 8.54668 61.8369 8.08001C61.6103 7.61335 61.2969 7.24668 60.8969 6.98001C60.4969 6.71335 60.0303 6.58001 59.4969 6.58001C58.9636 6.58001 58.4969 6.71335 58.0969 6.98001C57.7103 7.24668 57.4036 7.61335 57.1769 8.08001C56.9636 8.54668 56.8569 9.08668 56.8569 9.70001C56.8569 10.3267 56.9636 10.8733 57.1769 11.34C57.4036 11.8067 57.7103 12.1733 58.0969 12.44C58.4969 12.7067 58.9636 12.84 59.4969 12.84Z" fill="currentColor"></path></svg></a></span></div><ul class="flex items-center gap-4"><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-1 text-sm font-medium text-slate-11 hover:text-slate-12 focus-visible:text-slate-12 lg:px-2" href="/about">About</a></li><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-1 text-sm font-medium text-slate-11 hover:text-slate-12 focus-visible:text-slate-12 lg:px-2" href="/blog">Blog</a></li><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-1 text-sm font-medium text-slate-11 hover:text-slate-12 focus-visible:text-slate-12 lg:px-2" href="/customers">Customers</a></li><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-1 text-sm font-medium text-slate-11 hover:text-slate-12 focus-visible:text-slate-12 lg:px-2" href="/pricing">Pricing</a></li><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-1 text-sm font-medium text-slate-11 hover:text-slate-12 focus-visible:text-slate-12 lg:px-2" href="/enterprise">Enterprise</a></li><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-1 text-sm font-medium text-slate-11 hover:text-slate-12 focus-visible:text-slate-12 lg:px-2" href="/changelog">Changelog</a></li><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-1 text-sm font-medium text-slate-11 hover:text-slate-12 focus-visible:text-slate-12 lg:px-2" href="/docs">Docs</a></li></ul><div class="flex gap-4"><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 text-sm h-10 pl-4 pr-4 gap-0 font-semibold bg-slate-1 border-slate-1 text-slate-11 hover:bg-slate-5 hover:text-slate-12 focus-visible:ring-4 focus-visible:ring-slate-7 focus-visible:outline-none focus-visible:bg-slate-6 disabled:hover:bg-slate-1 inline-flex items-center border justify-center select-none rounded-full disabled:cursor-not-allowed disabled:opacity-70 transition ease-in-out duration-200 hidden lg:flex" data-state="closed" href="/login">Sign in</a><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 text-sm h-10 pl-4 pr-2 gap-0 font-semibold bg-white text-black hover:bg-white/90 focus-visible:ring-4 focus-visible:ring-white/30 focus-visible:outline-none focus-visible:bg-white/90 disabled:hover:bg-white inline-flex items-center border justify-center select-none rounded-full disabled:cursor-not-allowed disabled:opacity-70 transition ease-in-out duration-200" data-state="closed" href="/signup">Get Started<span class="text-[#70757E]"><svg fill="none" height="24" viewBox="0 0 24 24" width="24" xmlns="http://www.w3.org/2000/svg"><path d="M10.75 8.75L14.25 12L10.75 15.25" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5"></path></svg></span></a></div></div></div></header><div><div class="space-y-8 py-16 md:max-w-7xl mx-auto w-full max-w-5xl px-6"><p class="sans mt-6 text-center text-sm leading-[1.6] text-slate-11 font-normal"><time dateTime="2024-02-22">February 22nd, 2024</time></p><div><h2 class="font-book font-styling font-display mt-4 text-center text-[3rem] md:text-[3.5rem] tracking-tight leading-[120%] font-gradient"><span data-br=":R56kj6:" data-brr="1" style="display:inline-block;vertical-align:top;text-decoration:inherit;text-wrap:balance">Incident report for February 21st, 2024</span><script>self.__wrap_n=self.__wrap_n||(self.CSS&&CSS.supports("text-wrap","balance")?1:2);self.__wrap_b=(e,n,t)=>{t=t||document.querySelector(`[data-br="${e}"]`);let r=t.parentElement,o=f=>t.style.maxWidth=f+"px";t.style.maxWidth="";let c=r.clientWidth,a=r.clientHeight,s=c/2-.25,i=c+.5,p;if(c){for(o(s),s=Math.max(t.scrollWidth,s);s+1<i;)p=Math.round((s+i)/2),o(p),r.clientHeight===a?i=p:s=p;o(i*n+c*(1-n))}t.__wrap_o||(typeof ResizeObserver!="undefined"?(t.__wrap_o=new ResizeObserver(()=>{self.__wrap_b(0,+t.dataset.brr,t)})).observe(r):process.env.NODE_ENV==="development"&&console.warn("The browser you are using does not support the ResizeObserver API. Please consider add polyfill for this API to avoid potential layout shifts or upgrade your browser. Read more: https://github.com/shuding/react-wrap-balancer#browser-support-information"))};self.__wrap_n!=1&&self.__wrap_b(":R56kj6:",1)</script></h2><p class="sans text-center text-base md:text-[1.125rem] md:leading-[1.5] text-slate-11 font-normal"><span data-br=":R96kj6:" data-brr="1" style="display:inline-block;vertical-align:top;text-decoration:inherit;text-wrap:balance">Detailed postmortem of the outage on February 21st.</span><script>self.__wrap_n=self.__wrap_n||(self.CSS&&CSS.supports("text-wrap","balance")?1:2);self.__wrap_b=(e,n,t)=>{t=t||document.querySelector(`[data-br="${e}"]`);let r=t.parentElement,o=f=>t.style.maxWidth=f+"px";t.style.maxWidth="";let c=r.clientWidth,a=r.clientHeight,s=c/2-.25,i=c+.5,p;if(c){for(o(s),s=Math.max(t.scrollWidth,s);s+1<i;)p=Math.round((s+i)/2),o(p),r.clientHeight===a?i=p:s=p;o(i*n+c*(1-n))}t.__wrap_o||(typeof ResizeObserver!="undefined"?(t.__wrap_o=new ResizeObserver(()=>{self.__wrap_b(0,+t.dataset.brr,t)})).observe(r):process.env.NODE_ENV==="development"&&console.warn("The browser you are using does not support the ResizeObserver API. Please consider add polyfill for this API to avoid potential layout shifts or upgrade your browser. Read more: https://github.com/shuding/react-wrap-balancer#browser-support-information"))};self.__wrap_n!=1&&self.__wrap_b(":R96kj6:",1)</script></p></div><div class="mx-auto mb-10 mt-6 flex flex-row items-center justify-center gap-2"><span class="relative block flex-shrink-0 overflow-hidden rounded-full" style="background:linear-gradient(var(--background), var(--background)) padding-box, linear-gradient(42deg, var(--background) 45%, var(--slate-a10)) border-box;border:1px solid transparent"><img alt="Bu Kinoshita" fetchpriority="high" width="28" height="28" decoding="async" data-nimg="1" style="color:transparent" srcSet="/_next/image?url=%2Fstatic%2Favatars%2Fbu-kinoshita.jpg&w=32&q=75 1x, /_next/image?url=%2Fstatic%2Favatars%2Fbu-kinoshita.jpg&w=64&q=75 2x" src="/_next/image?url=%2Fstatic%2Favatars%2Fbu-kinoshita.jpg&w=64&q=75"/></span><span class="sans text-sm leading-[1.6] text-slate-11 font-normal">Bu Kinoshita</span></div><div class="col-span-2 overflow-hidden rounded-lg border border-slate-6 md:rounded-xl"><img alt="Incident report for February 21st, 2024" fetchpriority="high" width="1200" height="630" decoding="async" data-nimg="1" class="w-full object-cover" style="color:transparent" srcSet="/_next/image?url=%2Fstatic%2Fposts%2Fincident-report-for-february-21-2024.jpg&w=1200&q=100 1x, /_next/image?url=%2Fstatic%2Fposts%2Fincident-report-for-february-21-2024.jpg&w=3840&q=100 2x" src="/_next/image?url=%2Fstatic%2Fposts%2Fincident-report-for-february-21-2024.jpg&w=3840&q=100"/></div><div class="content mx-auto max-w-2xl"><h2>Summary (TL;DR)</h2> | |
| <p>On February 21st, 2024, Resend experienced an <a href="https://resend-status.com/incident/329826">outage</a> that affected all users due to a database migration that went wrong. This prevented users from using the API (including sending emails) and accessing the dashboard from 05:01 to 17:05 UTC (about 12 hours).</p> | |
| <p>The database migration accidentally deleted data from production servers. We immediately began the restoration process from a backup, which completed 6 hours later. Unfortunately, once it finished, we found that it failed to restore the data, so we had to start the restoration process a second time with a different backup.</p> | |
| <p>During this time, no API requests were being accepted and no data being stored. For data created prior to the migration, there was 5 minute of data loss from when the migration started and the database went offline from 04:50:00 to 04:56:27 UTC. We are currently working on re-populating the data from this 5-minute window.</p> | |
| <p>We sincerely apologize for the impact and inconvenience caused by this outage. We place immense importance on reliability, but this week, we fell short of our commitment to you all. It is clear that we have a long way to go in becoming an industry-leading infrastructure provider, but in learning from this incident, we will improve our operations and tooling to avoid outages like this in the future, whatever the cause.</p> | |
| <h2>Timeline</h2> | |
| <p><em>All times are in Coordinated Universal Time (UTC)</em></p> | |
| <p><strong>February 21st, 2024</strong></p> | |
| <ul> | |
| <li>04:56: Database migration started</li> | |
| <li>04:57: Noticed tables being dropped from the production database</li> | |
| <li>05:01: Began restoring the database from a backup</li> | |
| <li>05:02: Posted on <a href="https://resend-status.com/incident/329826">status page</a>, updating every 30-60 minutes until resolution</li> | |
| <li>11:02: First restoration process completed</li> | |
| <li>11:03: Realized the first backup failed and started to investigate</li> | |
| <li>11:33: Found that the backup failed due to a wrong selection of the backup timestamp</li> | |
| <li>11:48: Increased compute to speed up the restoration process - updated database memory from 128GB to 256GB and CPU from 32-core ARM to 64-core ARM</li> | |
| <li>12:05: Began restoring the database from an older backup</li> | |
| <li>17:01: Second restoration process completed</li> | |
| <li>17:02: API began receiving requests</li> | |
| <li>17:05: Dashboard was accessible again, and incident was resolved</li> | |
| </ul> | |
| <h2>What happened</h2> | |
| <p>While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production.</p> | |
| <p>The first attempt to restore the database took 6 hours but failed due to a wrong selection of the backup timestamp. The second attempt to restore took an extra 5 hours and succeeded, bringing all data back besides a 5-minute window of data loss.</p> | |
| <h2>Impact</h2> | |
| <p>All users were unable to send emails, use the API, or access the Resend dashboard for 12 hours from 05:01 to 17:05 UTC.</p> | |
| <p>For data created prior to the migration, there was 5 minutes of data loss from when the migration started and the database went offline from 04:50:00 to 04:56:27 UTC.</p> | |
| <h2>Next steps and improvements</h2> | |
| <ul> | |
| <li> | |
| <p>Re-populate data from the 5-minute window of data loss.</p> | |
| </li> | |
| <li> | |
| <p>No accessible user role should have write privileges on the production database.</p> | |
| </li> | |
| <li> | |
| <p>Improve local development to reduce risks related to database migrations.</p> | |
| </li> | |
| <li> | |
| <p>Create redundancy to preserve sending function even during a database outage.</p> | |
| </li> | |
| <li> | |
| <p>Increase cadence for disaster recovery tests.</p> | |
| </li> | |
| <li> | |
| <p>Implement incident banner on Resend dashboard to inform users quickly.</p> | |
| </li> | |
| </ul> | |
| <p>To our customers, we are deeply sorry that this incident occurred and that it prevented you from delivering your mission-critical emails. We know that actions speak louder than words, so we will continue to learn, grow, and improve, starting by implementing the improvements listed above.</p> | |
| </div></div><footer class="mx-auto max-w-5xl px-6 py-36 md:max-w-7xl"><div class="flex w-full flex-col gap-8 sm:flex-row"><div class="flex w-full flex-row sm:w-1/3 sm:flex-col"><div class="flex w-full flex-col gap-8 sm:mb-8"><p class="text-xs text-slate-11 font-normal">2261 Market Street #5039<br/>San Francisco, CA 94114</p></div><div class="flex w-full gap-4"><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-full" rel="noreferrer" target="_blank" href="https://x.com/resend"><img alt="Twitter" loading="lazy" width="32" height="32" decoding="async" data-nimg="1" style="color:transparent" src="/static/footer-twitter.svg"/></a><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-full" rel="noreferrer" target="_blank" href="https://github.com/resend"><img alt="GitHub" loading="lazy" width="32" height="32" decoding="async" data-nimg="1" style="color:transparent" src="/static/footer-github.svg"/></a><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-full" rel="noreferrer" target="_blank" href="https://linkedin.com/company/resend"><img alt="LinkedIn" loading="lazy" width="32" height="32" decoding="async" data-nimg="1" style="color:transparent" src="/static/footer-linkedin.svg"/></a><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-full" rel="noreferrer" target="_blank" href="/discord"><img alt="Discord" loading="lazy" width="32" height="32" decoding="async" data-nimg="1" style="color:transparent" src="/static/footer-discord.svg"/></a></div></div><div class="grid w-full grid-cols-2 gap-8 sm:w-2/3 lg:grid-cols-4"><div class="flex flex-col gap-4"><p class="mb-2 ml-1.5 text-sm text-slate-12 font-normal">Documentation</p><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/docs/introduction">Getting Started</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/docs/api-reference/introduction">API Reference</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/docs/integrations">Integrations</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/docs/examples">Examples</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/docs/sdks">SDKs</a></li></ul></div><div class="flex flex-col gap-4"><p class="mb-2 ml-1.5 text-sm text-slate-12 font-normal">Resources</p><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/changelog">Changelog</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/pricing">Pricing</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/security">Security</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/status">Status</a></li></ul></div><div class="flex flex-col gap-4"><p class="mb-2 ml-1.5 text-sm text-slate-12 font-normal">Company</p><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/blog">Blog</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/contact">Contact</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/customers">Customers</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/brand">Brand</a></li></ul></div><div class="flex flex-col gap-4"><p class="mb-2 ml-1.5 text-sm text-slate-12 font-normal">Legal</p><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/legal/acceptable-use">Acceptable Use</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/legal/privacy-policy">Privacy Policy</a></li></ul><ul><li><a class="outline-none transition duration-150 ease-in-out focus-visible:ring-2 focus-visible:ring-slate-7 rounded-md px-1 py-0.5 text-sm text-slate-11 outline-none hover:text-slate-12 focus-visible:text-slate-12" href="/legal/terms-of-service">Terms of Service</a></li></ul></div></div></div></footer></div></div></div><div role="region" aria-label="Notifications (F8)" tabindex="-1" style="pointer-events:none"><ol tabindex="-1" class="fixed bottom-0 right-0 z-50 flex w-[90%] gap-3 p-6 md:w-[390px]"></ol></div><svg height="0" viewBox="0 0 0 0" width="0"><defs><linearGradient gradientUnits="userSpaceOnUse" id="resend-dark-gradient" x1="13.5" x2="13.5" y1="11.328" y2="31"><stop stop-color="#E5E5E5"></stop><stop offset="1" stop-color="#535353"></stop></linearGradient></defs></svg></div><script id="__NEXT_DATA__" type="application/json" crossorigin="">{"props":{"pageProps":{"title":"Incident report for February 21st, 2024","slug":"incident-report-for-february-21-2024","description":"Detailed postmortem of the outage on February 21st.","image":"/static/posts/incident-report-for-february-21-2024.jpg","date":"2024-02-22","author_name":"Bu Kinoshita","author_avatar":"/static/avatars/bu-kinoshita.jpg","content":"\u003ch2\u003eSummary (TL;DR)\u003c/h2\u003e\n\u003cp\u003eOn February 21st, 2024, Resend experienced an \u003ca href=\"https://resend-status.com/incident/329826\"\u003eoutage\u003c/a\u003e that affected all users due to a database migration that went wrong. This prevented users from using the API (including sending emails) and accessing the dashboard from 05:01 to 17:05 UTC (about 12 hours).\u003c/p\u003e\n\u003cp\u003eThe database migration accidentally deleted data from production servers. We immediately began the restoration process from a backup, which completed 6 hours later. Unfortunately, once it finished, we found that it failed to restore the data, so we had to start the restoration process a second time with a different backup.\u003c/p\u003e\n\u003cp\u003eDuring this time, no API requests were being accepted and no data being stored. For data created prior to the migration, there was 5 minute of data loss from when the migration started and the database went offline from 04:50:00 to 04:56:27 UTC. We are currently working on re-populating the data from this 5-minute window.\u003c/p\u003e\n\u003cp\u003eWe sincerely apologize for the impact and inconvenience caused by this outage. We place immense importance on reliability, but this week, we fell short of our commitment to you all. It is clear that we have a long way to go in becoming an industry-leading infrastructure provider, but in learning from this incident, we will improve our operations and tooling to avoid outages like this in the future, whatever the cause.\u003c/p\u003e\n\u003ch2\u003eTimeline\u003c/h2\u003e\n\u003cp\u003e\u003cem\u003eAll times are in Coordinated Universal Time (UTC)\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFebruary 21st, 2024\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e04:56: Database migration started\u003c/li\u003e\n\u003cli\u003e04:57: Noticed tables being dropped from the production database\u003c/li\u003e\n\u003cli\u003e05:01: Began restoring the database from a backup\u003c/li\u003e\n\u003cli\u003e05:02: Posted on \u003ca href=\"https://resend-status.com/incident/329826\"\u003estatus page\u003c/a\u003e, updating every 30-60 minutes until resolution\u003c/li\u003e\n\u003cli\u003e11:02: First restoration process completed\u003c/li\u003e\n\u003cli\u003e11:03: Realized the first backup failed and started to investigate\u003c/li\u003e\n\u003cli\u003e11:33: Found that the backup failed due to a wrong selection of the backup timestamp\u003c/li\u003e\n\u003cli\u003e11:48: Increased compute to speed up the restoration process - updated database memory from 128GB to 256GB and CPU from 32-core ARM to 64-core ARM\u003c/li\u003e\n\u003cli\u003e12:05: Began restoring the database from an older backup\u003c/li\u003e\n\u003cli\u003e17:01: Second restoration process completed\u003c/li\u003e\n\u003cli\u003e17:02: API began receiving requests\u003c/li\u003e\n\u003cli\u003e17:05: Dashboard was accessible again, and incident was resolved\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2\u003eWhat happened\u003c/h2\u003e\n\u003cp\u003eWhile building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production.\u003c/p\u003e\n\u003cp\u003eThe first attempt to restore the database took 6 hours but failed due to a wrong selection of the backup timestamp. The second attempt to restore took an extra 5 hours and succeeded, bringing all data back besides a 5-minute window of data loss.\u003c/p\u003e\n\u003ch2\u003eImpact\u003c/h2\u003e\n\u003cp\u003eAll users were unable to send emails, use the API, or access the Resend dashboard for 12 hours from 05:01 to 17:05 UTC.\u003c/p\u003e\n\u003cp\u003eFor data created prior to the migration, there was 5 minutes of data loss from when the migration started and the database went offline from 04:50:00 to 04:56:27 UTC.\u003c/p\u003e\n\u003ch2\u003eNext steps and improvements\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eRe-populate data from the 5-minute window of data loss.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eNo accessible user role should have write privileges on the production database.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eImprove local development to reduce risks related to database migrations.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eCreate redundancy to preserve sending function even during a database outage.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eIncrease cadence for disaster recovery tests.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eImplement incident banner on Resend dashboard to inform users quickly.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTo our customers, we are deeply sorry that this incident occurred and that it prevented you from delivering your mission-critical emails. We know that actions speak louder than words, so we will continue to learn, grow, and improve, starting by implementing the improvements listed above.\u003c/p\u003e\n"},"__N_SSG":true},"page":"/blog/[slug]","query":{"slug":"incident-report-for-february-21-2024"},"buildId":"XSe6_RzH2FbEfcRaaL7by","isFallback":false,"gsp":true,"scriptLoader":[]}</script></body></html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment