January 2017

No incidents reported for this month.

December 2016

Outbound SMS Delays
All outbound messages have been delivered. Sorry for the issue today.
Dec 21, 14:31-14:55 GMT
Accessing the Service is Slow
Sorry about the slow patch, the issue has been resolved.
Dec 16, 12:20-12:27 GMT
Application Latency
This incident has been resolved.
Dec 8, 12:34-12:58 GMT
Full Text Search Unavailable
This incident has been resolved.
Dec 1, 10:56-14:29 GMT

November 2016

Full Text Search Unavailable
This incident has been resolved.
Nov 30, 16:40-16:52 GMT
Uploading Files to Emails or Templates
This incident has been resolved.
Nov 11, 11:11 - Nov 12, 09:35 GMT
Some Requests Resulting in Errors
This incident has been resolved.
Nov 11, 22:13-22:36 GMT

October 2016

File Uploads
This incident has been resolved.
Oct 6, 09:22-10:05 GMT

September 2016

Service Disruption
This issue was caused by a PSU (power supply) fault to a core router, along with a related hardware fault simultaneously, preventing normal failover behaviour, and leaving core routing in an abnormal state. The core routers at fault are due for replacement in the next few weeks, and our hosting provider will be assessing this incident carefully, to reduce the likelihood of something similar occurring again. Although the affected equipment is currently 'at risk' because the failed PSU has not yet been replaced, we're marking the incident as resolved as we don't expect any further problems. Sorry for the issue this afternoon, and thanks for your patience whilst we fixed it.
Sep 27, 12:42-13:31 GMT
Service Disruption
This issue was caused by severe packet loss across our services in Manchester and York, due to a router hardware fault. The offending equipment has been disabled. Sorry for the problems this morning.
Sep 15, 08:30-09:33 GMT

August 2016

SMS Not Sending
All messages have cleared through the system. We're very sorry for the delays this afternoon. Just to be clear, no messages were lost, only delayed.
Aug 30, 14:09-15:14 GMT
Service Outage
This incident has been resolved.
Aug 10, 03:33-03:42 GMT

July 2016

Service Disruption
This incident has been resolved.
Jul 28, 13:50-14:01 GMT
Keyword Search Problems
This incident has been resolved.
Jul 26, 11:17-12:38 GMT
Background Processing Delays
The backlog is now cleared, we're resolving the incident.
Jul 6, 13:20-14:14 GMT

June 2016

No incidents reported for this month.

May 2016

Background Processing Delays
This incident has been resolved.
May 16, 09:22-17:24 GMT

April 2016

Service Interruption
The incident is now resolved. In case you didn't get our prior update, reports are available again. We're leaving the task reminders in the menu bar disabled until late this evening. We will be writing a postmortem to explain what happened here, please check back by the end of the day to read. It will be published here. We know that uptime is crucial to you, so it's our first priority too. Sorry to have put you through these problems yesterday.
Apr 7, 12:45 - Apr 8, 07:31 GMT
Workflow Delays
Workflows are caught up and this incident is now resolved. Thanks for bearing with us whilst we fixed this.
Apr 4, 14:47-15:13 GMT

March 2016

Service Disruption
This incident has been resolved.
Mar 17, 13:31-14:17 GMT

February 2016

Some results not appearing when searching
This incident has been resolved.
Feb 24, 09:43-10:38 GMT

January 2016

No incidents reported for this month.

December 2015

[Scheduled] Scheduled Maintenance - 9pm Sunday
The scheduled maintenance has been completed.
Dec 6, 21:00-21:31 GMT
Incoming Email Delays
Our inbound email service provider has resolved the issue, all email has now been routed. If this affected you today, we're sorry about the hassle.
Dec 3, 13:02-20:36 GMT

November 2015

Performance Degraded
This incident has been resolved.
Nov 30, 11:41-13:07 GMT
Connections Timing Out
This incident has been resolved.
Nov 25, 07:53-09:20 GMT
Service Unavailable
Sorry for the trouble this afternoon, this is resolved.
Nov 19, 16:19-16:36 GMT
Keyword Search Delays
The problem is now resolved. Sorry if it impacted you. A failure in our monitoring systems compounded the situation too, which we understand and will work to resolve.
Nov 19, 11:19-11:35 GMT

October 2015

Service Unavailable
This incident has been resolved.
Oct 20, 22:35-22:46 GMT
Retrospective: Attaching local files to emails
Users would have seen an error message when trying to upload local file attachments to emails (not from file manager), or when attempting to upload files to the file manager over this weekend. This has now been resolved, and we're sorry if this impacted your work.
Oct 11, 13:53 GMT
Retrospective: Degraded access to files
Over the last week, two separate incidents have had an impact on our file cluster. This has caused slower than usual access to emails, templates and file attachments. Knock on effects of these incidents have been having an impact on service quality at various times of day. We've now put in place measures that will fix this issue, and apologise for the frustration this has caused you.
Oct 6, 11:43 GMT

September 2015

Degraded Service
This incident has been resolved.
Sep 29, 11:28-11:58 GMT
Problems Uploading/Attaching Files
Apologies if this caused you problems this lunchtime, all fixed now.
Sep 25, 13:13-13:37 GMT
Some Cached Data Being Displayed
This incident has been resolved.
Sep 22, 15:44-15:49 GMT
Retrospective: Sub leads referred between 11:30 and 12:00 BST.
An issue has affected delivery of sub leads between 11:30 and 12:00 BST today (22nd September). During this time, sub leads were referred, but lead actions weren't carried out. So you may find that the status hasn't changed, emails haven't triggered, etc.. To see if this might have affected you, please go to 'Search Leads', choose 'Date Range' of today, click 'Options' on the LHS filter options, click 'Search using referral date' and 'Search sub leads', and you'll then be able to see all the sub leads that were referred between 11:30 and 12:00. If necessary, you'll need to alter the status and send any emails that are missing. Sorry about this, and we hope it hasn't impacted you too much. If you need any help, please give us a call.
Sep 22, 14:36 GMT
Workflow Delays
This incident has been resolved.
Sep 8, 15:25-15:58 GMT

August 2015

Problems Saving or Updating Data
This incident has been resolved.
Aug 14, 03:58-04:33 GMT

July 2015

Service Unavailable
This incident has been resolved.
Jul 29, 11:23-16:01 GMT
Keyword search is not being updated
This incident has been resolved.
Jul 27, 17:39-18:36 GMT
Service Access Disrupted
We're sorry if this affected you. Service was unavailable for around 2 minutes. We've identified the cause and we're now marking this incident as resolved.
Jul 24, 15:24-15:32 GMT
Some cached data being displayed
Thanks for your patience whilst we fixed this. We're now closing the incident.
Jul 7, 12:11-14:14 GMT

June 2015

Some users reporting slow access
This incident has been resolved.
Jun 23, 16:06-19:23 GMT
Service is currently unavailable
This incident has been resolved.
Jun 22, 22:53-23:01 GMT
Connection Problems
The issue is resolved now. We're sorry to hold you up this afternoon.
Jun 18, 14:51-16:04 GMT
Full Text Search Delays
Full text searches are now returning up-to-date results again. Sorry for the hiccup.
Jun 2, 22:11-22:25 GMT

May 2015

No incidents reported for this month.

April 2015

Bounces and Failures on Outgoing Emails
We'll be emailing customers with more information about this shortly.
Apr 29, 13:48-18:58 GMT
Inbound Text Messaging
Our text messaging people have informed us that this is now fixed.
Apr 2, 16:27-17:18 GMT

March 2015

Inbound SMS on Shortcode 80080
This incident has been resolved.
Mar 27, 08:50-09:23 GMT
Outbound Delivery Delays
The issue has been fixed. There was an issue with one of our email platform's sending turrets that caused it to be unable to connect to any MX servers. All messages that temporarily failed will be retried and processed as normal. Sorry for the hassle this caused you this afternoon.
Mar 25, 16:10-17:10 GMT

February 2015

Background Job Processing Delays
This incident has been resolved.
Feb 27, 14:34-15:12 GMT

January 2015

Slow access to the service
We're really sorry for keeping you waiting for your emails today, although I can confirm access is back to normal.
Jan 30, 10:23-14:15 GMT
Vodafone SMS issues
This incident has been resolved.
Jan 19, 16:16-20:19 GMT
Inbound/outbound email delays
We're now marking this incident as resolved.
Jan 19, 11:38-19:08 GMT
Some connections timing out
We're closing this issue as things are looking good now. Sorry if this caused you a problem using the service earlier.
Jan 18, 19:43-21:30 GMT
Background processing delays
All looks well, this incident is now resolved.
Jan 18, 07:47-08:01 GMT

December 2014

Delays opening up emails
All is still well so we're closing this incident now. We'll make sure it doesn't reoccur.
Dec 4, 11:45-13:20 GMT
Waiting for emails to open
This issue cleared up this morning, and we think it's a long running backup on a fileserver that is causing the problem. We'll be running some tests this evening and keeping a close eye on the backup tomorrow. We should be able to react fast enough for this not to cause the same problem again tomorrow, but will update with a new incident if it does.
Dec 2, 10:03-16:14 GMT
Workflows with actions that happen when a lead is created
We're sorry about this problem this afternoon - it's now fixed.
Dec 1, 16:30-16:54 GMT
Degraded Service
We're now marking this incident as resolved.
Dec 1, 09:59-12:43 GMT

November 2014

Access Problems
We've now identified the root cause and will close the incident. Sorry for the hiccup.
Nov 28, 16:14-16:49 GMT
Notifications Out of Sync
This incident has been resolved.
Nov 18, 09:20-12:19 GMT
Access Problems
This problem is resolved. We don't believe access was affected, but users may have seen a patch of degraded service.
Nov 14, 15:11-15:17 GMT
Degraded Service
The problem has now been resolved. Sorry for the less than acceptable performance today.
Nov 13, 12:43-17:36 GMT

October 2014

Workflow Processing Delayed
This incident has been resolved.
Oct 17, 13:33-14:46 GMT
Access Problems
This is now resolved, sorry for the problems. It was caused be a DDoS (not directly against us, but affecting us via the network).
Oct 1, 13:28-13:54 GMT

September 2014

Access Problems
We've confirmed the root cause of the outage, but our hosting provider is currently looking into the circumstances behind it. We'll consider the issue resolved, and re-open a new incident if we see any further problems. Sorry for the problems today.
Sep 30, 11:46-12:18 GMT
Inbound Text Messaging
We've now been informed that the problem has been fixed. Inbound messages queued during this incident are now being delivered.
Sep 24, 09:29-10:06 GMT
Degraded Service
The problem has been cleared now. We kept seeing positive signs just moments after noticing the problem that it was about to be resolved any moment, so we hesitated logging an incident.
Sep 23, 11:10-11:19 GMT

August 2014

Degraded Service
This incident has been resolved.
Aug 18, 15:39-18:29 GMT
Access Problems
We experienced power loss to a critical component within our network, which caused a cascade failure in our authentication system. This caused an endless loop of redirects as users tried to access the service. We've now fixed the fault, sorry to have interrupted your work this afternoon.
Aug 1, 14:51-15:05 GMT

July 2014

Lower Priority Background Task Delays
This incident has been resolved.
Jul 23, 12:52-13:15 GMT
Workflow Processing Behind
We've isolated the problem and workflow processing is now caught up. Sorry for the delays this morning.
Jul 4, 10:26-10:46 GMT
Full Text Search Delays
Thanks for your patience, we've fixed the problem.
Jul 1, 16:58-17:12 GMT

June 2014

Inbound Text Messages (O2) Disrupted
This incident has been resolved.
Jun 27, 14:31-19:33 GMT
Inbound Email Problems
Our service provider has told us that all systems are now back to normal. If they update again, we'll re-open this incident, but consider it resolved.
Jun 26, 12:32-13:50 GMT
554 Rejection on Incoming Email
Our service provider has told us the issue is resolved. If the situation changes we'll update you here. Our apologies if this impacted you.
Jun 16, 15:05-15:36 GMT
Emails Larger Than 500KB Delayed
Our email service provider have notified us that they resolved this problem.
Jun 4, 15:32-18:15 GMT
Access Problems
We've resolved the issue now. Our primary database failed, and to prevent data inconsistencies the cluster didn't automatically fail over to another server. We manually failed over, then had to complete some more work to bring the other servers back online with the new master. It did mean that from the point where we brought the master back online (approx 5:17), it was operating on its own until a few minutes ago which meant service was very, very slow. Sorry for the disruption this afternoon.
Jun 4, 16:37-18:14 GMT

May 2014

Low Priority Background Task Delay
The issue is resolved. The cause is linked to new hardware we're running - we need to make some changes to the way tasks are prioritised for this new hardware, which we'll do urgently. Note that this did not affect high priority tasks (i.e. new leads being processed).
May 21, 15:36-15:43 GMT
Inbound Lead Processing
Sorry for the delay this morning. As this is now resolved we're closing this incident.
May 19, 10:27-11:19 GMT
Some Requests Failing
This incident has been resolved.
May 13, 11:24-13:35 GMT
Degraded Service
Service is running normally.
May 8, 10:26-11:28 GMT
Access problems
We have found the error and resolved the issue. We're sorry we went away for a few minutes there.
May 2, 14:30-14:33 GMT

April 2014

Degraded Service
We fixed a problem with one of our database servers that was causing some connections to hang. Service is running smoothly again now.
Apr 28, 08:15 GMT
Connection Problems
We're sorry for the outage - we recovered from a database error.
Apr 16, 23:19-23:23 GMT
Slow Service
Service is almost back to 100% now. We'll post an incident report here once we can explain more about the incident and our next steps. Thank you.
Apr 15, 10:20-16:14 GMT
Outgoing Email
Our ESP hasn't officially confirmed it yet, but we can see from our own monitoring that emails are no longer delayed. If the situation changes again, we'll update here.
Apr 14, 16:34-17:12 GMT
Jobs Queue - Degraded Performance
All remaining jobs are now in progress. Apologies for today's delays and slow queues.
Apr 10, 12:44-16:44 GMT
Service Disruption
This incident has been resolved.
Apr 8, 15:07-15:18 GMT

March 2014

No incidents reported for this month.