Results 1 to 21 of 21

Thread: Crystal PM " Security Update" Allservers down until Friday...

  1. #1
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300

    Redhot Jumper Crystal PM " Security Update" Allservers down until Friday...

    Yikes!

    Per recorded message to support phone line it looks like we are dead in the water until Friday 1pm EST.

  2. #2
    Master Jedi King of the Lab's Avatar
    Join Date
    Jul 2012
    Location
    Los Angeles, CA.
    Occupation
    Optical Laboratory Technician
    Posts
    413
    Wait what?!

    My Crystal system is running fine.....
    Erik Zuniga, ABOC.

  3. #3
    What's up? drk's Avatar
    Join Date
    Mar 2004
    Location
    Ohio
    Occupation
    Optometrist
    Posts
    9,385
    Is it a cloud-based? I'm server based and it's working..

  4. #4
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300
    Quote Originally Posted by drk View Post
    Is it a cloud-based? I'm server based and it's working..
    It is apparently cloud based. The back office and Doc's kept running all day but they were told they will not be able to log back in, after they shut down shortly, until tomorrow sometime.

  5. #5
    What's up? drk's Avatar
    Join Date
    Mar 2004
    Location
    Ohio
    Occupation
    Optometrist
    Posts
    9,385
    Live by the cloud, die by the cloud...

    But it could be far, far worse.

  6. #6
    Master OptiBoarder
    Join Date
    Jun 2012
    Location
    Mitten State
    Occupation
    Ophthalmic Technician
    Posts
    713
    and this is why I have reservations regarding operating completely from the cloud.

  7. #7
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300

  8. #8
    What's up? drk's Avatar
    Join Date
    Mar 2004
    Location
    Ohio
    Occupation
    Optometrist
    Posts
    9,385
    As most of you know, Crystal PM cloud servers have been temporarily disabled due to an influx of ransomware attacks. On 12/03/20, after the shut down, it was discovered that ransomware had been introduced into the datacenter where Crystal servers are hosted. Crystal PM uses Netgain, a major secure data center, to handle the hosting of all of our online servers. We chose them because they are one of the biggest and most secure data centers in the United States. They have security staff 24 hours monitoring and maintaining their servers.

    At this time, it is not believed to have affected any Crystal PM servers. However, due to this, extra precautions are needing to be taken and multiple servers are needing to be recreated, which will extend the amount of time necessary for the system to be offline. Our initial estimate was 12pm CST on Friday, 12/04/2020. At this time, we are now being told that the system should be back online by Monday, 12/07/2020.

    Please be aware, this has not only affecting our cloud servers, but is occurring worldwide. Amazon's AWS servers were recently taken offline for a similar issue. Hospitals and schools server systems are currently being locked down due to attacks as well. While CrystalPM and Netgain do everything in our power to try to prevent these problems, new methods for distribution and attack are being developed every day.

    If you are not a cloud office, you may also be affected by what is happening.
    • You may not be able to import patients from web. While this service is not completely shut down, due to the amount of traffic, it may be non-functional and may cause your system to lock up and freeze.



    • Freezing when printing invoices - this will occur if your system is configured to print the patient portal information on the invoice. To prevent this, please go to Admin - Invoices and uncheck the option to "Include information on how to access the portal.



    • Freezing when opening CrystalPM or going to the tasks page – please go to Admin – Tasks and make sure the checkbox to “Auto Check Web Messages” is unchecked.






    • You are not able to upload documents to the patient portal until this is resolved



    • eDr contact lens orders cannot be sent until this is resolved



    • The VisionWeb integration will not function properly until this is resolved



    • Schedule Your Exam will be down until the issue is resolved.
    Again, we sincerely apologize for the inconvenience this is causing and are doing everything in our power to get everyone back up and running as quickly as possible.

  9. #9
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300
    self delete

  10. #10
    Master OptiBoarder
    Join Date
    Jun 2012
    Location
    Mitten State
    Occupation
    Ophthalmic Technician
    Posts
    713
    Quote Originally Posted by drk View Post
    As most of you know, Crystal PM cloud servers have been temporarily disabled due to an influx of ransomware attacks. On 12/03/20, after the shut down, it was discovered that ransomware had been introduced into the datacenter where Crystal servers are hosted. Crystal PM uses Netgain, a major "secure" data center.
    Fixed that.

  11. #11
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300
    Well it's still down so I assume Crystal will work on some type of compensation for those of us still waiting to enter our work for the last few days.

  12. #12
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300
    Still dead in the water.

    They're saying a few servers up later today.. or maybe tomorrow...this time for sure......no really.......

    The chances of the office going paperless are now greatly diminished.

    The latest FAQ from Crystal PM and some of the others affected:

    Are Just Crystal PM offices affected?
    • No, all cloud servers at Netgain are currently offline. Netgain hosts cloud servers for Hospitals, several other EHR companies, health systems, post-acute providers, medical device and drug manufacturers, insurers, and medical groups.

  13. #13
    What's up? drk's Avatar
    Join Date
    Mar 2004
    Location
    Ohio
    Occupation
    Optometrist
    Posts
    9,385
    This is cyber terrorism.

    Pretty frightening if you think about it.

  14. #14
    OptiBoard Novice
    Join Date
    Sep 2015
    Location
    Columbus OHIO
    Occupation
    Optometrist
    Posts
    1
    NetGain is a small player on the server farm playing field. AWS and Microsoft are by far the largest. Crystal is not written for the cloud. They are a hosted system with data at the users site and data at the server farm. you are putting your data at great risk by using a hosted system instead of a true cloud system. CrystalPM is trying to minimize their mistakes by not investing in technology and rewriting their system. Saying hospitals and schools are down is a copout. I have not heard of any hospitals down this wee and trust me, we would be hearing about it. Most hospitals are on EPIC, which is a server based system. Please hold those guys accountable for not taking care of your data. I just hope your data isnt compromised because then you are responsible for those violations.

  15. #15
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300
    Thanks for your post and welcome to Optiboard!

    And I mean it sincerely when I say I'm sorry we had to meet this way!

    Still adrift in the doldrums until this weekend we're told.

    Oh to go back to the days of my youth when the order was on a 3x5 index card.

  16. #16
    What's up? drk's Avatar
    Join Date
    Mar 2004
    Location
    Ohio
    Occupation
    Optometrist
    Posts
    9,385
    Kkurt, a fellow Columbusite with the BIIIIGGGGG post.

    I don't know you, but I like you a lot.

  17. #17
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300
    Good thing it's not a busy time of year.

    Cloud Status Update - Monday Afternoon

    Crystal PM is still waiting the login credentials for the servers from Netgain for the 1st batch of cloud servers. We have confirmed that 65% of our offices will be included in that batch and that they are online and many are already sending data to 3rd party systems including Schedule Your Exam. Schedule Your Exam and Crystal PM Online Forms \ Portals are included in that batch as well and went online last night.

    The current issue is the newly created domain controllers and creating user accounts for the servers. Netgain has escalated a trouble ticket to the highest level and Crystal PM is eagerly standing by at the moment we get access to set passwords and begin notifying offices.

    For the doctors concerned with data breach. The attack was against Netgain and their infrastructure. Authorities have been notified. Netgain informed us there is an official forensic file that they will share with us once Netgain has all severs restored that shows zero files were encrypted or transferred from any Crystal PM server. We will send this file to offices when available.

  18. #18
    What's up? drk's Avatar
    Join Date
    Mar 2004
    Location
    Ohio
    Occupation
    Optometrist
    Posts
    9,385
    Torture.

  19. #19
    OptiBoard Novice
    Join Date
    Oct 2020
    Location
    south dakota
    Occupation
    Dispensing Optician
    Posts
    3
    Our office has been unable to access Officemate since December 3. We use Netgain servers. What a nightmare. Those in the same situation have you gotten any updates on when we might be able to access again.

  20. #20
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300
    We came back on line Wednesday about 3pm when our office manager tried to open it. So much for them contacting us! Unfortunately after being shut down for yesterdays storm we came back today to find it not working.

    Promises made are no longer believed so the work keeps piling up.

    Staying away from high places and sharp objects!



    You may, like us, want to try to open it every couple of hours just to see what happens.

  21. #21
    Compulsive Truthteller OptiBoard Gold Supporter Uncle Fester's Avatar
    Join Date
    Dec 2004
    Location
    At a position without dimension...
    Occupation
    Dispensing Optician
    Posts
    5,300
    It's working!

    Now to enter the stacks of jobs......

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Some Friday Fun.... Is the "customer always right "?
    By eryn in forum General Optics and Eyecare Discussion Forum
    Replies: 13
    Last Post: 11-04-2014, 03:38 PM
  2. Just substitute "eyeglasses" or "OTCs" for "umbrellas"
    By Barry Santini in forum General Optics and Eyecare Discussion Forum
    Replies: 29
    Last Post: 08-28-2011, 01:27 PM
  3. Terrorism update: al-Qaida's "Geek Squad" second to none
    By rinselberg in forum Just Conversation
    Replies: 3
    Last Post: 03-31-2008, 09:17 AM
  4. Replies: 0
    Last Post: 10-24-2006, 01:12 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •