HomeGuidesBlueprintsCommunityChangelogAPI ReferenceGet support
Guides

Elasticsearch migration: 7.x to 9.5

How to move your Onify Hub API from Elasticsearch 7.x to 9.5. There are three methods, each in its own section. Choose one and follow only that section.

Section 1: In-place upgrade

  • Use when: You want to keep one cluster
  • Version path: 7.x → 7.17.29 → 8.19.24 → 9.5.6
  • Hub API during the migration: Stopped
  • Way back: Restore a snapshot into a new cluster

Section 2: New cluster

  • Use when: You can run a second cluster that reaches the current one
  • Version path: 7.14 or later → 9.5.6
  • Hub API during the migration: One instance, with no user traffic
  • Way back: Point the Hub API at the old cluster

Section 3: Snapshot

  • Use when: You can run a second cluster, but it cannot reach the current one
  • Version path: 7.x → temporary 8.19.24 → 9.5.6
  • Hub API during the migration: Stopped
  • Way back: Point the Hub API at the old cluster

Required versions

  • Elasticsearch: 9.5.6, or the newest 9.5.x patch. Do not use 9.1, 9.2 or 9.3: Elastic has
    no fix for ESA-2026-189
    on those lines.
  • If you use a newer 8.19.x patch than 8.19.24, use a 9.5.x patch released the same day or later.
  • Hub API and workers: the same version, 2.51.3 or later. The Hub API does not run on Elasticsearch 8.x, or below 7.14.

Before you start

  • Run the whole procedure on a copy of your production data first, and use the timing to plan your maintenance window.
  • Contact Onify first if you run a multi-node cluster, multitenant mode or very large indices.
  • Add a snapshot repository before the window, because it needs a node restart: set path.repo and mount a backup volume.
  • The commands need bash, curl and jq, and use Docker Compose with services named elasticsearch, hub-api and
    hub-worker. Add -u <user> to curl on a cluster with a password.

Set these in your shell. Every section uses them:

export PREFIX=onify                              # ONIFY_db_indexPrefix
export HUB_API=http://localhost:8181/api/v2
export TOKEN=$(printf 'app:%s' "$ONIFY_apiTokens_app_secret" | base64 | tr -d '\n')

# Document count per index
counts() {
  for index in $(curl -fsS "$1/_cat/indices/$PREFIX-*?h=index&s=index"); do
    printf '%s %s\n' "$index" "$(curl -fsS "$1/$index/_count" | jq -r .count)"
  done
}

# Creation version of the indices
created() {
  curl -fsS "$1/$PREFIX-*/_settings?human&filter_path=*.settings.index.version.created_string" \
    | jq -c '[.[].settings.index.version.created_string] | group_by(.) | map({(.[0]): length}) | add'
}

# Read and write through the Hub API. Expected: UP, create 201, read 200, delete 200
test_hub_api() {
  curl -fsS "${HUB_API%/api/v2}/health" | jq -r .status
  local type; type=$(curl -fsS "$1/_cat/indices/$PREFIX-item-*?h=index&s=index" | head -n 1 | sed "s/^$PREFIX-item-//")
  curl -fsS -X POST "$HUB_API/admin/items" -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' \
    -d "{\"key\":\"migration-probe\",\"name\":\"Migration probe\",\"type\":\"$type\",\"tag\":[\"migration\"]}" \
    -o /dev/null -w 'create %{http_code}\n'
  sleep 2
  curl -fsS "$HUB_API/admin/items/migration-probe" -H "Authorization: Bearer $TOKEN" -o /dev/null -w 'read %{http_code}\n'
  curl -fsS -X DELETE "$HUB_API/admin/items/migration-probe" -H "Authorization: Bearer $TOKEN" -o /dev/null -w 'delete %{http_code}\n'
}

After any migration

  • Take a first snapshot on 9.5, and set up regular backups.

  • Keep the old cluster or first snapshot until your agreed observation period ends.

  • If you use GET /tools/logs?pagination=cursor on the Hub API, set this on the 9.5 cluster. Elastic has deprecated the
    setting.

    curl -fsS -X PUT "<elasticsearch-url>/_cluster/settings" -H 'content-type: application/json' \
  -d '{"persistent":{"indices.id_field_data.enabled":true}}'

Section 1: In-place upgrade

The existing cluster goes through each version on the same data, so no second cluster is needed. The Hub API stays stopped
until the end, because it cannot run on 8.x.

7.x → 7.17.29 → 8.19.24 → rebuild indices → 9.5.6

Elasticsearch cannot be downgraded. Once you start, the only way back is the snapshot in step 1.1, and anything written
after it is lost.

Prepare

Add this before your maintenance window, then restart the node:

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION}
    environment:
      - discovery.type=single-node
      - path.repo=/usr/share/elasticsearch/backup
      - xpack.security.enabled=false   # keep your current choice, but set it explicitly
    volumes:
      - es-data:/usr/share/elasticsearch/data
      - es-backup:/usr/share/elasticsearch/backup
docker compose exec -u 0 elasticsearch chown elasticsearch:root /usr/share/elasticsearch/backup
export ES=http://localhost:9200

upgrade_to() {
  docker compose stop elasticsearch
  ES_VERSION=$1 docker compose up -d --force-recreate elasticsearch
  until curl -fsS "$ES/_cluster/health?wait_for_status=yellow&timeout=60s" >/dev/null 2>&1; do sleep 2; done
  curl -fsS "$ES" | jq -r .version.number
}

If upgrade_to does not finish, read docker compose logs elasticsearch.

1.1 Stop the Hub API and take a snapshot

docker compose stop hub-api hub-worker
counts "$ES" > before-counts.txt
curl -fsS -X PUT "$ES/_snapshot/before-upgrade" -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/backup/before-upgrade"}}'
curl -fsS -X PUT "$ES/_snapshot/before-upgrade/pre-upgrade?wait_for_completion=true" -H 'content-type: application/json' \
  -d "{\"indices\":\"$PREFIX-*\",\"include_global_state\":false}" | jq -c '.snapshot | {state, shards}'

Expected: "state":"SUCCESS" and "failed":0. Add any other indices you need to indices.

1.2 Upgrade to 7.17.29

Skip upgrade_to if you already run 7.17.29.

upgrade_to 7.17.29
curl -fsS "$ES/_migration/deprecations?pretty"

Expected: 7.17.29. Resolve every item with level critical before you continue. For example, an index created before 7.0
must be reindexed or deleted.

1.3 Upgrade to 8.19.24

upgrade_to 8.19.24
curl -fsS "$ES/_migration/system_features" | jq -r .migration_status

Expected: 8.19.24, then a status. If the status is MIGRATION_NEEDED, migrate the system indices and wait for it to finish:

curl -fsS -X POST "$ES/_migration/system_features" | jq -c '{accepted}'
until [ "$(curl -fsS "$ES/_migration/system_features" | jq -r .migration_status)" = NO_MIGRATION_NEEDED ]; do sleep 3; done

1.4 Rebuild the indices

9.5 cannot use indices created in 7.x, so each one is rebuilt here. Save this as rebuild.sh:

#!/usr/bin/env bash
set -euo pipefail
: "${ES:?set ES to the 8.19.x URL}" "${PREFIX:?set PREFIX to the index prefix}"
CT='content-type: application/json'
definition() {
  curl -fsS "$ES/$1" | jq --arg i "$1" '.[$i] | {
    settings: {index: (.settings.index | del(.uuid, .version, .provided_name, .creation_date, .routing, .resize, .blocks, .history))},
    mappings
  }'
}
count() { curl -fsS "$ES/$1/_count" | jq -r .count; }
copy() {
  local failures
  failures=$(curl -fsS -X POST "$ES/_reindex?wait_for_completion=true&refresh=true" -H "$CT" \
    -d "{\"source\":{\"index\":\"$1\"},\"dest\":{\"index\":\"$2\"}}" | jq '.failures | length')
  [ "$failures" = 0 ] && [ "$(count "$1")" = "$(count "$2")" ] || { echo "MISMATCH $1 -> $2" >&2; exit 1; }
}
for index in $(curl -fsS "$ES/_cat/indices/$PREFIX-*?h=index&s=index"); do
  case "$index" in *-reindexed) continue ;; esac
  def=$(definition "$index")
  curl -fsS -X PUT "$ES/$index-reindexed" -H "$CT" -d "$def" >/dev/null
  copy "$index" "$index-reindexed"
  curl -fsS -X DELETE "$ES/$index" >/dev/null
  curl -fsS -X PUT "$ES/$index" -H "$CT" -d "$def" >/dev/null
  copy "$index-reindexed" "$index"
  curl -fsS -X DELETE "$ES/$index-reindexed" >/dev/null
  echo "rebuilt $index: $(count "$index") documents"
done

Run it, then check the result:

bash rebuild.sh
created "$ES"
curl -fsS "$ES/_migration/deprecations" | jq -c '{indices_flagged: (.index_settings | length), flagged: (.index_settings | keys)}'
counts "$ES" > rebuilt-counts.txt
diff before-counts.txt rebuilt-counts.txt && echo "counts match"

Expected: an 8.19 version for every index, such as {"8.19.9-8.19.24":N}, then {"indices_flagged":0,"flagged":[]} and
counts match. Any other index still flagged also blocks 9.5, so reindex or delete it.

1.5 Upgrade to 9.5.6

Take a second snapshot, in a new location, then upgrade:

curl -fsS -X PUT "$ES/_snapshot/after-rebuild" -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/backup/after-rebuild"}}'
curl -fsS -X PUT "$ES/_snapshot/after-rebuild/rebuilt?wait_for_completion=true" -H 'content-type: application/json' \
  -d "{\"indices\":\"$PREFIX-*\",\"include_global_state\":false}" | jq -c '.snapshot | {state, shards}'
upgrade_to 9.5.6
counts "$ES" > after-counts.txt
diff before-counts.txt after-counts.txt && echo "counts match"

Expected: "state":"SUCCESS", 9.5.6 and counts match.

1.6 Start the Hub API and open traffic

Start Hub API 2.51.3 or later, with no user traffic:

docker compose up -d hub-api hub-worker
test_hub_api "$ES"

Expected: the Hub API and worker logs show Detected database version: 9.5.6 (client: es9), and the test prints UP,
create 201, read 200 and delete 200. Then open user traffic, and test a login and a real process.

If something goes wrong

  • Problem: A problem before step 1.2

    What to do: Nothing has changed. Start the Hub API again

  • Problem: rebuild.sh stops

    What to do: Do not delete anything. Contact Onify, or go back

  • Problem: 9.5 exits with must be marked as read-only using the setting [index.blocks.write]

    What to do: Do not add that setting: the Hub API could never write to those indices. Run upgrade_to 8.19.24, and repeat step 1.4

  • Problem: Hub API logs Accept version must be either version 8 or 7

    What to do: It was started on 8.x. Keep it stopped until step 1.6

  • Problem: A problem after users wrote to 9.5

    What to do: Fix it on 9.5. Going back loses everything written since the snapshot

To go back: stop the Hub API and the cluster, and start a new, empty cluster on your old 7.x version, on a new data
volume, with the same backup volume. Restore the snapshot, then start your old Hub API version:

curl -fsS -X PUT "$ES/_snapshot/before-upgrade" -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/backup/before-upgrade","readonly":true}}'
curl -fsS -X POST "$ES/_snapshot/before-upgrade/pre-upgrade/_restore?wait_for_completion=true" -H 'content-type: application/json' \
  -d "{\"indices\":\"$PREFIX-*\",\"include_global_state\":false}" | jq -c .snapshot.shards
counts "$ES" > restored-counts.txt
diff before-counts.txt restored-counts.txt && echo "counts match"

Section 2: New cluster

An empty 9.5.6 cluster runs next to the current one. One Hub API call creates each index on the new cluster, and the new
cluster pulls the documents over HTTP. The current cluster stays as it is, so it remains your way back until users write to
the new one. No 8.x version is needed.

7.14 or later → 9.5.6

Prepare

  • If you run below 7.14, upgrade to 7.17.29 first, as in step 1.2.
  • Upgrade the Hub API and workers to 2.51.3 or later while they are on the current cluster.
  • The new cluster must reach the current one using the exact host and port in ONIFY_db_elasticsearch_host, and the
    Hub API must reach the new cluster.
  • Plan disk for your primary data times (1 + replicas), and stay below 85% disk use.
  • Only <prefix>-* indices are copied. Elasticsearch users and roles, Kibana, ILM policies and other indices stay behind.
  • New indices take their shards and replicas from the Hub API setting db.indexSettings (default 1 shard, 0 replicas). Set
    ONIFY_db_indexSettings_numberOfReplicas on the Hub API before the copy if you need replicas.

2.1 Build the new cluster

services:
  target-es:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.6
    environment:
      - discovery.type=single-node
      - reindex.remote.whitelist=source-es:9200   # the host and port from ONIFY_db_elasticsearch_host
      - ES_JAVA_OPTS=-Xms4g -Xmx4g
      - xpack.security.enabled=false              # remove for HTTPS, see "Secured clusters"
    volumes:
      - es9-data:/usr/share/elasticsearch/data
    ports:
      - 9201:9200
volumes:
  es9-data:
export SRC=http://localhost:9200             # the current cluster, from this shell
export DST=http://localhost:9201             # the new cluster, from this shell
export DST_FOR_HUB_API=http://target-es:9200 # the new cluster, from the Hub API
curl -fsS "$DST" | jq -r .version.number
curl -fsS "$DST/_cat/indices/$PREFIX-*?h=index" | wc -l

Expected: 9.5.6 and 0.

2.2 Stop writes

From here on, users cannot change anything.

  1. Stop all workers.
  2. Stop all Hub API replicas except one, and take that one out of your ingress or load balancer.
  3. Block writes on every index except the two that Hub API instance still needs:
for index in $(curl -fsS "$SRC/_cat/indices/$PREFIX-*?h=index&s=index"); do
  case "$index" in "$PREFIX-instance"|"$PREFIX-log") continue ;; esac
  curl -fsS -X PUT "$SRC/$index/_settings" -H 'content-type: application/json' -d '{"index.blocks.write": true}' >/dev/null
done

2.3 Take a snapshot and record the counts

In Docker, make the backup volume writable once:

docker compose exec -u 0 elasticsearch chown elasticsearch:root /usr/share/elasticsearch/backup
curl -fsS -X PUT "$SRC/_snapshot/migration" -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/backup"}}'
curl -fsS -X PUT "$SRC/_snapshot/migration/pre-migration?wait_for_completion=true" -H 'content-type: application/json' \
  -d "{\"indices\":\"$PREFIX-*\",\"include_global_state\":false}" | jq -c '.snapshot | {state, shards}'
counts "$SRC" > source-counts.txt

Expected: "state":"SUCCESS" and "failed":0.

2.4 Copy the data

deleteIndices clears any index with your prefix on the new cluster, so only point this at the new, empty cluster.

curl -sS -X POST "$HUB_API/setup/database/upgrade" -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d "{\"destination\":{\"host\":\"$DST_FOR_HUB_API\",\"deleteIndices\":true}}" | jq -c '{message, statusCode}'

Expected: "statusCode":200. This only means the copy started. The result is in the Hub API log:

[remote-reindex] successfully reindexed onify-role 30 document(s) in 89ms
[remote-reindex] Remote reindex completed
Upgrade finished with status 200

Continue only on status 200. On any other status, fix the cause in the log and run the same command again, for all indices.

If the call answers Indexes (<name>) not found, that index has no definition in your Hub API version. Copy the others by
adding ?indices=<comma-separated names> to the call, and handle that index yourself if you need it.

2.5 Verify

counts "$DST" > target-counts.txt
diff source-counts.txt target-counts.txt && echo "counts match"
created "$DST"
curl -fsS "$DST/_cluster/health?filter_path=status,unassigned_shards"

Expected: counts match, {"9.5.6":N}, and a status of green or yellow. Also compare a few important records between
the clusters. Do not continue until this is clean.

2.6 Switch the Hub API and open traffic

  1. Stop the Hub API instance.
  2. Set ONIFY_db_elasticsearch_host to the new cluster on the Hub API and workers.
  3. Start them with no user traffic, and run test_hub_api "$DST".

Expected: the logs show Detected database version: 9.5.6 (client: es9), and the test prints UP, create 201, read 200
and delete 200. Then open user traffic, and test a login and a real process.

This is the point of no return: data written to 9.5 cannot be copied back to 7.x. Leave the old cluster's write blocks on,
and remove reindex.remote.whitelist from the new cluster.

If something goes wrong

Until users write to the new cluster, you can go back:

  1. Stop the Hub API and workers.

  2. Remove the write blocks:

    curl -fsS -X PUT "$SRC/$PREFIX-*/_settings" -H 'content-type: application/json' -d '{"index.blocks.write": null}'

  3. Point ONIFY_db_elasticsearch_host back to the old cluster, and start the Hub API and workers.

After users write to the new cluster, fix the problem there. Going back loses everything written since the switch.

Secured clusters

If the current cluster has a password, give the Hub API instance a settings file, and name it in the ENV_PATH variable,
relative to the application folder:

db.elasticsearch.auth.username=<user>
db.elasticsearch.auth.password=<password>
db.elasticsearch.username=<user>
db.elasticsearch.password=<password>

If the new cluster uses HTTPS, which is the default in Elastic's image:

  1. Add hostname: target-es to the service before its first start, and a volume for
    /usr/share/elasticsearch/config. The certificate must cover the name the Hub API uses.

  2. Get the password and the CA certificate:

    docker compose exec target-es bin/elasticsearch-reset-password -u elastic -b
docker compose cp target-es:/usr/share/elasticsearch/config/certs/http_ca.crt .

  3. Use https://, and add --cacert http_ca.crt -u elastic to curl commands on the new cluster.

  4. Copy http_ca.crt to /usr/app/data on the Hub API and workers, and set
    NODE_EXTRA_CA_CERTS=/usr/app/data/http_ca.crt. Recreate the containers so that it takes effect.

  5. In step 2.4, add "username":"elastic","password":"<password>" inside destination.

  6. In step 2.6, set ENV_PATH to a file with the two db.elasticsearch.auth.* lines above, for the new cluster.

Section 3: Snapshot

Use this if the clusters cannot reach each other, but can share a snapshot volume. The data travels through a temporary
8.19.24 cluster, because 9.5 cannot restore indices created in 7.x. The Hub API stays stopped until the end. The current
cluster stays as it is, so your way back is to start the Hub API on it again.

7.x snapshot → restore on temporary 8.19.24 → rebuild → snapshot → restore on new 9.5.6

export SRC=http://localhost:9200      # the current cluster
export BRIDGE=http://localhost:9202   # the temporary 8.19.24 cluster
export DST=http://localhost:9201      # the new 9.5.6 cluster

3.1 Stop the Hub API and take a snapshot

Stop all Hub API replicas and workers, then:

docker compose exec -u 0 elasticsearch chown elasticsearch:root /usr/share/elasticsearch/backup
counts "$SRC" > source-counts.txt
curl -fsS -X PUT "$SRC/_snapshot/migration" -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/backup"}}'
curl -fsS -X PUT "$SRC/_snapshot/migration/pre-migration?wait_for_completion=true" -H 'content-type: application/json' \
  -d "{\"indices\":\"$PREFIX-*\",\"include_global_state\":false}" | jq -c '.snapshot | {state, shards}'

Expected: "state":"SUCCESS" and "failed":0.

3.2 Restore on a temporary 8.19.24 cluster

Start an empty 8.19.24 cluster with the backup volume and a second, empty volume. List both paths in path.repo separated by
a comma, set xpack.security.enabled explicitly, and make the second volume writable with chown as in step 3.1.

curl -fsS -X PUT "$BRIDGE/_snapshot/migration" -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/backup","readonly":true}}'
curl -fsS -X POST "$BRIDGE/_snapshot/migration/pre-migration/_restore?wait_for_completion=true" -H 'content-type: application/json' \
  -d "{\"indices\":\"$PREFIX-*\",\"include_global_state\":false}" | jq -c .snapshot.shards

Expected: "failed":0.

3.3 Rebuild the indices and snapshot them

Save rebuild.sh from step 1.4, then:

ES=$BRIDGE PREFIX=$PREFIX bash rebuild.sh
created "$BRIDGE"
curl -fsS -X PUT "$BRIDGE/_snapshot/bridge" -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/bridge"}}'
curl -fsS -X PUT "$BRIDGE/_snapshot/bridge/reindexed?wait_for_completion=true" -H 'content-type: application/json' \
  -d "{\"indices\":\"$PREFIX-*\",\"include_global_state\":false}" | jq -c '.snapshot | {state, shards}'

Expected: an 8.19 version for every index, such as {"8.19.9-8.19.24":N}, then "state":"SUCCESS" and "failed":0. Then
stop the temporary cluster.

3.4 Restore on the new 9.5.6 cluster

Start an empty 9.5.6 cluster with the second volume, listed in path.repo.

curl -fsS -X PUT "$DST/_snapshot/bridge" -H 'content-type: application/json' \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/bridge","readonly":true}}'
curl -fsS -X POST "$DST/_snapshot/bridge/reindexed/_restore?wait_for_completion=true" -H 'content-type: application/json' \
  -d "{\"indices\":\"$PREFIX-*\",\"include_global_state\":false}" | jq -c .snapshot.shards
counts "$DST" > target-counts.txt
diff source-counts.txt target-counts.txt && echo "counts match"
created "$DST"

Expected: "failed":0, counts match, and an 8.19 version for every index. These are normal, writable indices.

3.5 Start the Hub API and open traffic

Set ONIFY_db_elasticsearch_host to the new cluster on Hub API and workers 2.51.3 or later. Start them with no user traffic,
and run test_hub_api "$DST".

Expected: the logs show Detected database version: 9.5.6 (client: es9), and the test prints UP, create 201, read 200
and delete 200. Then open user traffic, and test a login and a real process.

If something goes wrong

  • Problem: Any problem before user traffic

    What to do: The current cluster has not changed. Start the Hub API on it again with the old database host

  • Problem: rebuild.sh stops

    What to do: Delete the temporary cluster's data, and start again from step 3.2

  • Problem: 9.5 answers cannot restore index, or must be marked as read-only

    What to do: The snapshot still holds 7.x indices. Restore the one from step 3.3, and do not add the setting

  • Problem: A restore reports failed shards

    What to do: Check that the disk is below 85% use

  • Problem: A problem after users wrote to 9.5

    What to do: Fix it on 9.5. Going back loses everything written since the switch

Updated 1 day ago