Meet Ultipa Manager: Easy Data Migration

In general terms, data migration is the process to move data from one source platform, system, location or format to another. The scope of data migration work varies a lot in terms of volume (hundreds, millions, or billions), quality (poorly structured, repetitive, or well-structured, clean) and status (static, or still growing in real-time) of data, as well as types of the source and target (whether they are similar or can be easily converted). We rarely meet business clients who like to start from scratch, data migration is always topping their laundry list. In this article, we will discuss how Ultipa Manager offers you intuitive ways to facilitate data migration, and how some interesting functionalities can be boosted via extra Ultipa products.

As you may already know, Ultipa Manager is our Graph Database Management System (GDBMS, or Graph DBMS). This is the last post in the Meet Ultipa Manager series. In the past two weeks, we’ve covered the topics on graph visualization and toolkits for data scientists .

Understanding Graph Data Structure

It is necessary to understand the data structure of graph before conducting any data migration involving Ultipa Graph database. Graph is made up of nodes (aka. vertices) and edges; each edge must be attached to two distinct or identical nodes. In Ultipa Graph, all edges are directed, the two end nodes of each edge are called the start node and the end node respectively. In real-world context, nodes are mainly connected with other nodes by edges, though some may exist by themselves alone, which we call isolated nodes. Isolated nodes usually don't have much analytical significance.

Whether node or edge, it’s structured in schema and property. Schema is the type, or label that each data must carry, represented by symbol @ in Ultipa Graph; each schema contains some properties that describe the data in different aspects. Below is an example of 3 node schemas and 2 edge schemas, the properties and example values are shown next to each schema.

Example of creating schema, properties and nodes by UQL , Ultipa’s graph query language:?

a. Create node schema @user:

create().node_schema("user")        

b. Create properties of @user (Note: property _id is the unique identifier created along with schema by default):

create()
  .node_property(@user, "username", string)
  .node_property(@user, "level", uint32)        

c. Create two @user nodes:

insert().into(@user).nodes([
  {_id: "U256", username: "joe23", level: 2},
  {_id: "U257", username: "mattYoung", level: 3}
])        

This is not the key point of this article, but these UQL commands are sometimes very convenient to add a small amount of data.?

File Import and Export in Ultipa Manager

In many cases, data to be migrated is delivered in CSV format. In Ultipa Manager, nodes and edges can be imported from or exported to CSV files (note that additional file formats can be supported, and we chose to use CSV to illustrate Ultipa Manager’s functionalities here). The file import in Ultipa Manager allows you to load data incrementally without changing historical or untouched data, while the database is kept online. The import speed can be easily surpassing 300,000 pieces of data per second.?

Each CSV file to be imported should contain only the information of nodes or edges of one schema. Below is the screenshot of 3 CSV files, from left to right are for node schemas @individual, @company and edge schema @holdShare. The optional first row of each file is the property names separated by comma with the sequence corresponds to the content below.

Ultipa Manager supports to import one file each time, this is the upload entry:

Next, configure the uploaded file by choosing the intended schema and making sure all properties are matched. For more configuration details, please refer to the Ultipa Manager User Guide .

Before executing batch-import, you can select to import data in the way of insert, upsert or overwrite.

An import record is provided for you to double check what was performed. Any errors will be skipped during the import, meanwhile they are clearly counted, saved and explained in the record; other necessary information is also displayed for reference.

The file export is more straightforward. You are allowed to export only the selected or all schemas and properties of node or edge. Similarly, nodes or edges of different schemas will be saved in separated CSV files.

Ultipa Transporter: Highly Efficient

Ultipa Transporter is a command-line-based tool for extremely fast data import and export in Ultipa Graph database. You can either operate remotely (Go version; MacOS, Windows, Linux) or locally (C++ version; Docker).

With the Importer of Ultipa Transporter, nodes and edges can be packed in many formats – CSV, TSV, TXT, etc.; in addition, you need to prepare a configuration file (e.g. import.yml); then executing import with only one command:

./ultipa-importer --config ./import.yml        

With the Exporter, only the configuration file is needed. Refer to Ultipa Transporter User Guides for detailed instructions.

In comparison with the file import/export in Ultipa Manager, Ultipa Transporter comes with faster data processing speed and flexibilities including:

• No need to create the graphset and schemas before performing the import.
• All node and edge files can be imported at one time.
• Nodes and edges can be exported at the same time.
• More file formats are supported.

Ultipa Maker: Data Migration in ETL

Ultipa Maker is a web-based data migration tool capable of transferring data from rich and diversified sources to Ultipa Graph database. Currently, the supported data sources are: Ultipa, Neo4j, Hive, MySQL, CSV and DAT. Database source (Ultipa, Neo4j, Hive, MySQL) is added by configuring its hosting server and the authorized credential; file source (CSV, DAT) is add by uploading the file and specifying the format.

An ETL pipeline involves Extract, Transform and Load steps:

1. Extract data from a database, a file, an API, etc.
2. Transform the extracted data from its original form to a proper format that can be mapped to the target platform.
3. Load the transformed data into the target platform. In Ultipa Maker, the target platform is Ultipa Graph database.

ETL is the smallest unit of data migration project. In Ultipa Maker, you can create as many ETLs as needed in one project, each ETL is meant for the nodes or edges import of one schema. You can arrange the operation order of multiple ETLs with sufficient rules, and schedule the operation by scheduler or trigger.

This is a migration project in Ultipa Maker, three ETLs have been created for importing @movie, @country nodes and @filmedIn edges (left pane). Now the first ELT is selected, on the right pane, the top two cards are the data source and target Ultipa Graph database respectively, below is the data projection of it. Data projection is to match the data column in the source with the property name in Ultipa Graph.

User guides of Ultipa Maker can be found here .

Closing Remarks

The file import/export in Ultipa Manager is favored by business personnel especially due to its user-friendly interface. Some users use the Manager to import really large data fields, say, graph dataset with 100s of millions of nodes and edges. However, for maximum productivity, it’s recommended to use more dedicated toolkits like Transporter. Ultipa Transporter executes the import/export more efficiently with a command line fashion. Ultipa Maker, furthermore, works with abundant data sources to satisfy broader needs. All of them share the ability of high speed and online incremental import without interrupting the database service. If you like to try Ultipa Transporter and Ultipa Maker, please contact us at [email protected] . Ultipa Manager is available in Ultipa Cloud , our DBaaS (Database as a service) platform.

I hope you enjoyed these 3 posts; any comments and discussions are warmly welcomed. Ultipa Manager is constantly undergoing updates, we’re always seeking to make it more powerful, helpful, and easy-to-use.