Metadata Migration in the JFrog Platform

Overview

The JFrog Metadata server is a JFrog Platformmicroservice that is used for storing, managing, and sharing metadata between different components of the JFrog Platform. The microserviceis part of JFrog Artifactory and is bundled with JFrog Artifactory 7.x installations. The microservice is used to store and sharemetadata for all packages from any JFrog services, such as Artifactory and JFrog Xray, and has its own configuration settings and logs files.

当从一个版本升级Artifactoryanother, the metadata must also be migrated to the new version. The length of this process will vary depending on the number of packages being migrated - as well as the number of versions being migrated.

This migration occurs automatically when upgrading from JFrog Artifactory 6.x to 7.x and requires no user action.

Fetching Metadata Using GraphQL

GraphQL is a query language that enables you to query your APIs and fetch specific data that you need.Using GraphQL to query metadata provides a simple way to fetch packages data stored in the metadata microservice. To learn more, seeGraphQL.

Viewing Metadata in the JFrog Platform UI

The Metadata server stores metadata, collected or calculated, from all the different JFrog products and services.

The Metadata data-model is used to manage the package-versions metadata, which can be consumed using various filters and search options through the JFrog platform UI - "Packages View". To learn more, seePackage Management.

Troubleshooting

To learn how to troubleshoot issues in the microservice, seeMetadata Service Troubleshooting.

Page Contents

Frequently Asked Questions on Metadata Migration

What happens to the UI during the migration?

When the migration takes place, the JFrog Platform UI displays a notification that the metadata is currently being migrated.

What happens if the migration stops in the middle?

The migration will continue from the same point the next time Artifactory is started.

Does the migration happen during or after the upgrade?

The migration occurs after the upgrade, and will be triggered on the first system initialization after the upgrade (the first time you start Artifactory v7.x).

Is there any downtime during migration?

JFrog tested the migration process using a benchmark ofArtifactory of 27k packages and 1.2M versions:

Artifactory ran with the default migration configurations (5 threads migrating to MDS).
The Artifactory and a PostgreSQL database were run on different machines with the following spec - GCP VM with 8 CPU & 30 Gb RAM.

There is no downtime caused by the migration, however, search results in the UI and in the GraphQL API will be limited.

How long will the migration take and what is the rate of conversion?

Based on the benchmark above, the migration rate was60k-67k versions per hour. The total migration using these specifications tookbetween 18-22 hours.

Database Details

Database Connections

The Metadata service uses a databaseconnection pool that by default contains 100 connections.Running Artifactory with the default MDS migration settings (5 threads migrating metadata), no significant raise in the active connections to the database (about 10 live connections) were detected.In the worst case, the number of connections will not pass 100 connections.

CPU

The PostgreSQL CPU max value was 60%.
The MDS CPU max value was 23%

RAM

The PostgreSQL RAM consumption was about 5%
The MDS RAM consumption was about 10%

Storage

The database size increased from 95GB to 155GB (~60% increase).

Managing the Metadata Migration Process

Customers can control the metadata migration by limiting the number of threads in Artifactory (threads<=1 means that the migration is off):

shared.extraJavaOpts -Dartifactory.metadata.event.operator.threads=5

Customers cancheck the migration status by querying the database as follows.

SELECT encode(migration_info_blob::bytea, 'escape') FROM migration_status where identifier = 'metadata-service-migration';

Content

Space Tools