Pre-Upgrade Activities

1. Planned and approved service window

   • Make sure that you have a planned and approved service window for the upgrade

2. Review Changelog

   • Make sure to review the changelogs for all versions from your current one up to the version you plan to upgrade to, checking for any breaking changes or re-indexing requirements in both the APP and API.
   • Reference: Onify Support Changelog

3. Check Available Disk Space

   • Ensure there is sufficient free disk space for the upgrade.
   • Check the size of existing indices using Elasticsearch commands.
   • Reference: How to calculate total index size in Elasticsearch

4. Ensure Availability of Snapshots

   • Confirm that a recent snapshot is available.
   • Recommended to use Elasticsearch snapshots.
   • Reference: How do I effectively set up and manage snapshots in Elasticsearch?

5. Set Up Scheduled Export of Onify Configurations

   • Implement a scheduled flow to export all configurations and resources from Onify.
   • Reference: How can I export configuration from Onify?

6. Activate Maintenance Mode

   • Enable maintenance mode to ensure no users are active during the upgrade.

Upgrade

1. Perform the Actual Upgrade
   • The upgrade process of Onify will depend on your specific implementation and on which container platform (Docker, Kubernetes, OpenShift etc) Onify is hosted. Onify might as well have been deployed using Infrastructure as Code (IaC), such as through tools like Terraform, necessitating specific routines related to IaC processes like running pipelines during the upgrade.
   • Please check our FAQ related to upgrade where we cover some of the scenarios: https://support.onify.co/discuss?tag=upgrade.

Post-Upgrade Activities

1. Execute Reindexing (if required)

   • If upgrade demands reindexing, process each index individually.
   • Ensure the number of documents reindexed matches the backed-up count.

2. Verify Scheduled Flows

   • Confirm that all scheduled flows and processes are running smoothly.

3. Verify Functionality

   • Ensure workspaces are functioning correctly post-upgrade.
   • Components to verify:
     • Bulletins
     • Shortcuts
     • Search Results / Item Cards
     • Forms
       • Initiate key processes from the forms to verify functionality.

4. De-activate Maintenance Mode

   • Turn off maintenance mode to allow user access to the system.

Troubleshooting

Reindexing

• If it's not possible to start the reindex process and you immediately get an "es_rejected_execution_exception" in the Elastic container log, check that the heap size for the elastic container is configured correctly. Should be set to half of total RAM of the node
  • e.g. ES_JAVA_OPTS=-Xms8g -Xmx8g if the node has a total of 16GB RAM
• In case of errors, use the exported configuration for restoration if possible, or revert to the snapshot and restore the indexes from the snapshot.
• Reference: Restore elasticsearch snapshots

Elastic container log after a successful reindex

2023-11-09 12:12:37 2023-11-09T11:12:37.615Z INFO [reindex] backed up onify-bulletin 4 document(s) in 103ms
2023-11-09 12:12:40 2023-11-09T11:12:40.582Z INFO [reindex] successfully reindexed onify-bulletin 4 document(s) in 70ms
2023-11-09 12:12:41 2023-11-09T11:12:41.188Z INFO Reindex completed

Elastic container log after a failed reindex

2023-10-24T17:39:52.016Z INFO [reindex] backed up onify-bulletin 27 document(s) in 51ms
2023-10-24T17:39:52.381Z INFO [reindex] successfully reindexed onify-bulletin 0 document(s) in 1ms
2023-10-24T17:39:52.433Z INFO Reindex completed

Disk space

• If the total amount of free disk space isn't enough for reindexing (at least the same amount as the current size of the indices requiring reindexing), consider managing the "process index" size. The "process index" often is the largest, storing the process and related data (variables, flowstate, BPMN process XML).
  • Purge variables and BPMN process XML for completed processes (e.g. processes started by users and completed more than 3 months ago) to save space while retaining the actual processes. Use this blueprint for configuring recurring cleanup of process variables and bpmn xml: Blueprint for purging variables and BPMN process XML from Onify processes (WIP).
    • NOTE: This requires version 2.14+ of Onify API
  • For processes like indexing processes, it's normal to completely remove them more frequently (e.g. daily) than processes started by users. Use this blueprint to configure purging of processes on schedule: Blueprint for purging Onify processes.
• If managing the "process index" size is not enough, increase the disk space to accommodate the upgrade and reindexing requirements.