Confirmations & campaigns incident
Incident Report for RevenueWell
Resolved
What’s Happening:
The fix that was implemented by our team appears to have resolved this issue.

Why This Matters:
All communications are sending & receiving as expected.

What Happens Next:
Our teams will continue to monitor all systems and processes, but at this time we are considering this incident resolved.
Posted Dec 01, 2022 - 07:44 CST
Monitoring
What’s Happening:
The fix that was implemented by our team last night appears to have resolved the root cause of the incident that began late last week. Incidents reported by customers today have been confirmed to be lingering symptoms of the incident that is now resolved.

Why This Matters:
Currently, all communications are sending & receiving as expected.

What Happens Next:
Our teams will continue to monitor all systems and processes that have been impacted over the last few days and will update this space as necessary.
Posted Nov 30, 2022 - 14:00 CST
Identified
Our engineers believe the issues sending patient communications and showing their status in the Practice Portal were resolved overnight. We are investigating ongoing reports of confirmations not writing back to Practice Management Systems.
Posted Nov 30, 2022 - 09:06 CST
Update
What's happening:
Today's Campaigns are not sending as required, and the team has identified what we believe to be the root cause. A fix has been implemented, and our system is now sending all relevant messages (ex., not sending same-day reminders to appointments that have passed). We anticipate that these messages will catch up by tonight and return to sending in real-time

Why this matters:
Campaigns that are still relevant will be sent.

What happens next:
Messages will return to sending in real-time, and we will continue to monitor this situation closely.
Posted Nov 29, 2022 - 15:57 CST
Monitoring
We are seeing progressive improvements and expect a full resolution mid-afternoon, possibly between 2-3pm CST.
Posted Nov 29, 2022 - 12:52 CST
Identified
We have discovered that alerts are not firing when patients ask for a call back (or send another unrecognized response to their appointment reminder). We are working to restore this as quickly as possible.

For 95% of customers, all confirmations have now written back and will continue to do so in real time. For the small set of customers using legacy Messenger, our Engineers continue to work on restoring this functionality.

Patients are continuing to receive the appropriate communications. However, the practice portal is not accurately showing which have been sent. We anticipate it will catch up around 1pm CST and accurately show all of the communications patients have received. Again - there has been no disruption to the messages patients are actually receiving.
Posted Nov 29, 2022 - 11:09 CST
Update
Patients are continuing to receive the appropriate communications. However, the practice portal is not accurately showing which have been sent. We anticipate it will catch up around 1pm CST and accurately show all of the communications patients have received. Again - there has been no disruption to the messages patients are actually receiving.

Additionally, there was a disruption over the weekend to confirmations writing back. This issue was resolved last night, and appointments confirmed today are writing back. Appointments confirmed between November 24 - 28 have not yet written back, and we are tirelessly working to make sure they do.

A next update will be provided around 1pm CST. We appreciate your patience as we work around the clock to resolve this incident.
Posted Nov 29, 2022 - 09:25 CST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 28, 2022 - 20:06 CST
Update
We are testing a fix to restore confirmation writeback and anticipate an update soon about its release. Our Engineers are also continuing to work on the practice portal not accurately showing which campaigns patients have received.
Posted Nov 28, 2022 - 15:54 CST
Update
We are continuing to work on a fix for this issue.
Posted Nov 28, 2022 - 12:33 CST
Identified
We have also identified that patient confirmations are not writing back to Practice Management Systems. We are actively working on the fix and will provide another update soon.
Posted Nov 28, 2022 - 09:58 CST
Update
We are continuing to investigate this issue.
Posted Nov 28, 2022 - 09:30 CST
Update
We have confirmed that patients are receiving their confirmations and other campaigns. However the Practice Portal is not reporting which messages have been sent. Another update will be provided within 1 hour.
Posted Nov 28, 2022 - 09:11 CST
Investigating
We are currently investigating an issue with confirmations and other campaigns not showing as sent in the Practice Portal. A next update will be provided in 1 hour or less.
Posted Nov 28, 2022 - 09:02 CST
This incident affected: Portals (Practice Portal) and Communications (Patient Communications).