Amazon Web Services Outage 2/28/17


#1

Having been unable to access my Canary devices for several hours I just checked the Canary Website where there’s a notice labeled “minor service outage.”

Amazon’s S3 service has crashed, taking down many service providers including Canary. I would consider that a very major problem for a security comapany. If Canary is shrugging this off, it gives me great concern.

Apparently they’re relying on this one vendor for everything. Is that really wise?


Gaps in recordings
#2

I too am having difficulty with Canary. It gives me a notice that it is offline even though everything else in the house is working fine.

If there was a notice of an outage coming I don’t know where I should see this. I spent time trying to reconnect with no success and that is when I decided to get on the computer and come to this forum.

When is this to come back on line?


#3

Also waitin for Canary to come back online :disappointed:


#4

To be clear, this was not scheduled. Amazon provides services to many other companies, and it was their servers that crashed today.

In addition to Canary I lost my Nest thermostats and smoke alarms. They’re now back up, so apparently some service is being restored.


#5

If you go to help.canary.is you’ll see the “minor service outage” banner that I referred to. Minor, unless someone chooses this moment to break into your house.


#6

Hey @Tangible,

Thanks for your feedback about this. We put up the status update page right away when we realized there was a problem, before we knew the full scope of how our services were being affected by Amazon S3’s outage. We have since upgraded the status to a Major outage.


#7

Will my canary app on the phone automatically reconnect to the camera? Or do I need to start all over in setting it up?

(As of this writing the app does not connect… I assume the all clear has not been announced)


#8

Also, what is the ETA for the restoration of service? And will we lose history as a result?


#9

@fullwave2 We do not currently have an ETA on the full restoration of Canary service. You will not lose your past video history, but there will be gaps in your timeline as a result of today’s Amazon outage. For more information and updates please check out our status page here: https://canary.statuspage.io/.


#10

@BobD @fullwave2 All services have been fully restored. The Canary app should automatically reconnect to the camera at this time. We appreciate your patience throughout this evening while our engineers worked through the residual effects of todays major AWS outage.


#11

Thanks for the updates, Matt. I hope Canary is giving some thought to the wisdom of leaving customers completely vulnerable to a system failure at a single supplier. AWS isn’t the only game in town, and it might be prudent for you to have some redundancy.


#12

@Matt Thanks for the update and for the quick recovery.
There is some value in the suggested redundant systems to insure a failsafe continuance of service if another occasion of loss of service occurs.
I’m sure you’ll be doing a cost versus reward analysis during your follow up meetings to determine what can be done to avoid a prolonged outage as we had today. To be validly considered a “security service” I think this will be necessary. Thanks again. Bob


#13

@Tangible @fullwave2 You are most welcome. Yesterdays events with AWS definitely revealed some challenges that many cloud based companies face as we rely heavily on Amazons services for parts of our underlying infrastructure, but it also presents an opportunity for us to continually work to improve our services. We’re sincerely grateful for your feedback and thanks again for raising this topic in our user community forum.


#14

It is after 3pm (cst), Wednesday, March 1 and my Canary has not been restored. I have sent numerous emails and have not received a reply. Please advise me when I can expect my service returned


#15

Hey @Mskitty2u,

Sorry for the late response! We’ve been catching up on tickets from yesterday. I found your ticket and I’ve sent you a response, so please feel free to continue troubleshooting with me via email and I’ll make sure we get you up and running :slight_smile:


#16

I did receive your email and have replied. What can I expect are next steps?


#17

@Matt, AFAIK disruptions to AWS was local to East region, at high level, does canary (client to AWS) not invest in BCP strategy to fall back on other colos and prevent such events? As a customer I would never would like to loose my home surveillance.

ref: https://aws.amazon.com/message/41926/


#18

A post was merged into an existing topic: Canary Status - “degraded performance”?


#19

No big deal it was just my smoke alarms…:confounded: