HIRO LogoHIRO Developer Portal
Getting Started
Tutorials
Overview
Documentation
API
Solutions API
7
  • Migration Topics
  • Integration
  • Csvconnector
    • About the csv connector
  • Concepts
  • Best Practices
  • Api
  • ActionHandler

About the csv connector

The csv connector ships with every HIRO instance and enables you to create Automation issues and upload environment and monitoring data. It makes use of HIROs Rest API using CSV files as input. We’ve put together sample csv files for each scenario to help you get started.

First things first: get your token!

A token is mandatory to use any of the capabilities the csv connector offers. To retrieve a token, you must run a Post request against https://core.arago.co/api/auth/6/app.

{
    "client_secret": "ffc55c735c79d22441e941b09fbe9afbda839dc12012b7bd37f9d88b54fd13add74fdef8abf8194e9729a281bfbe68935950fef7a9dd690827082fc611f68492",
    "client_id": "cju16o7cf0000mz77pbwbhl3q_ckkwosk448atc01543wvzxvb4",
    "username":"",
    "password": ""
}

Username and password is created for every HIRO 7 instance that is provisioned for you. The login information is generated and can be found on id.arago.co. The organisation admin will be able to change the pregenerated password for the user that is called haas*-csv-connector. The scope id, an id that identifies which HIRO 7 instance the csv content can be found doing a GET request with the token from the username above against https://core.arago.co/api/auth/6/me/account?profile=true The scope can be found in "ogit/Auth/Account/defaultScope"

How to use the CSV Connector

The CSV Connector works with HTTP requests. All capabilities share the same base url (https://csvconnector-881340721.eu-west-1.elb.amazonaws.com) followed by the action after the slash, such as /vertices /edges /timeseries or /attachments. You can either run those request from command line or any tool of your choice (such as postman for instance)

Please make sure to add the following header information to your requests

x-forwarded-authorization: Bearer ${your_token}
content-type: text/csv

Additional header parameters control details of csv handling within the connector:

Header Description

x-connector-csv-dialect

Set the csv dialect for the csv parser (default, unix, excel, excel-tab or excel-semicolon). Default is default, which is a version of unix with additional settings:
quoting=csv.QUOTE_ALL, skipinitialspace=True, strict=True.
The parser used internally is from Pythons' csv package.

x-connector-csv-json-strings

Boolean value. When set to true, turns on the interpretation of the CSV data cells as JSON strings. Default is false.

x-connector-csv-null-keyword

Take the string given here as null keyword to set attributes for HIRO explicitly to None.
Default keyword is null.

1. How to create ontology objects (vertices/nodes)

With the token in hand you can run a POST request against.

https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/vertices

using the following sample CSV file. Please make sure to add a value for scope. If you want to create more than one object, you can add more lines to the csv file. Keep in mind POST does not allow you to update existing objects. Please use a PUT request if you want to update.

"ogit/_xid","ogit/_type","ogit/MARS/Machine/class","ogit/name","ogit/MARS/Software/class","ogit/MARS/Software/subClass","ogit/_scope"
"machine1","ogit/MARS/Machine","Linux","machine a","","","cjuwixjvq0xfc1q90xqn8jsvf_ckcd798hd7bji0r882l764ebl"
"software1","ogit/MARS/Software","", "Software A","DBMS","MariaDB","cjuwixjvq0xfc1q90xqn8jsvf_ckcd798hd7bji0r882l764ebl"

2. How to inject an automation issue

With the token in hand you can run a POST request against https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/vertices

using the following sample CSV file. Please make sure to add a value for scope and origin node. The origin node defines where the issue is bound to first, where knowledge can be applied (if available). If you do not have an origin Node, please check part 1 where we show you how to create one.

"ogit/subject","ogit/Automation/originNode","/ProcessIssue","ogit/_type","ogit/_scope"
"This is a test Issue","node_id","process_me","ogit/Automation/AutomationIssue","scope_id"

3. How to create an edge (connect a Node)

With the token in hand you can run a POST request against https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/edges

using the following sample CSV file.

"from:ogit/_xid","verb","to:ogit/_xid"
"software1","ogit/dependsOn","machine1"

4. How to create an edge and node combined

With the token in hand you can run a PUT request against https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/combined

"ogit/_xid","ogit/_type","ogit/MARS/Machine/class","ogit/name","ogit/MARS/Software/class","ogit/MARS/Software/subClass","ogit/_scope","edge:0:verb","edge:0:direction","edge:0:vertex_xid"
"machine1","ogit/MARS/Machine","Linux","machine a","","","cjuwixjvq0xfc1q90xqn8jsvf_ckcd798hd7bji0r882l764ebl","","",""
"software1","ogit/MARS/Software","","Software A","DBMS","MariaDB","cjuwixjvq0xfc1q90xqn8jsvf_ckcd798hd7bji0r882l764ebl","ogit/dependsOn","out","machine1"

5. How to create a timeseries object

First create a timeseries node With the token in hand you can run a POST request against https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/vertices

"ogit/_xid","ogit/_type","ogit/name","ogit/_scope"
"timeseries","ogit/Timeseries","tsname","scopeid"

Then

With the token in hand you can run a PUT request against https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/timeseries

"ogit/_xid","timestamp","value"
"timeseries","1442235678000","45"

Error Handling

  • Feedback is provided if the csv file was processed successfully and how many objects of each type were populated.

  • A hard stop occurs incase the reading of a csv file encounters a malformed entry.

  • All entries before the malformed entry will be processed

Format of CSV data

Specifications of the format of csv data so the CSV connector can parse the data correctly. Parsing is based of Pythons' csv package.

General format

The encoding is UTF-8.

The separator is a comma ,.

Each row must have the same number of values. Rows must be filled with empty values if necessary.

Each value in the CSV must be enclosed in double-quotes ". Quotes are escaped via doubling, i.e. "I am ""quoted"".".

The CSV has to have a header row. For vertices these are the ogit attribute names.

The CSV will be transformed using Pythons' csv package with a special dialect called default (unless configured otherwise in header parameter x-connector-csv-dialect). The settings for this dialect are: quoting=csv.QUOTE_ALL, skipinitialspace=True, strict=True

Each row below the header specifies a set of values for one entity (vertex, edge, timeseries or attachment).

The strings parsed from each cell can be interpreted as JSON when header parameter x-connector-csv-json-strings is enabled. The values of the CSV will then be enclosed in quotes and run through json.loads(escaped_value). If this is set to true, special characters like \, ", line-breaks or tabulator characters need to be properly escaped for JSON and remember that you still have to quote the " for CSV as well.

Examples of a CSV cell

  • without JSON Strings:

"Hello ""friend"",
how are you? Look at C:\Users\Foo\"

  • with JSON Strings:

"Hello \""friend\"",\nhow are you? Look at C:\\Users\\Foo\\"

Format of the CSV data for vertices

Relevant REST endpoints:

POST https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/vertices
PATCH https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/vertices
PUT https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/vertices
DELETE https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/vertices

The header row contains ogit attribute names.

An empty value "" in a row tells the connector to ignore the attribute specified by that column for this row instead of setting it to an empty value.

A string value of "null" (or whatever is currently configured in header parameter x-connector-csv-null-keyword) in a row will make the connector set the attribute specified by that column explicitly to JSON null. This is needed to remove attributes of a vertex on an update.

Headers that start with xid: and contain ogit/_xid values are resolved to ogit/_id by querying HIRO.

Example:
xid:ogit/Automation/originNode would treat all values below as ogit/_xid and resolve them to ogit/_id before using them as attribute ogit/Automation/originNode in the vertices to create or update.

Headers that start with =, + or - denote map value entries (see HIRO attribute specification). Values within these columns are directly interpreted as JSON.

Example:

"ogit/_xid","ogit/_type","ogit/MARS/Machine/class","ogit/name","ogit/MARS/Machine/ram","/admin_contact"
"machine4","ogit/MARS/Machine","Linux","machine 4","2G","info@admin.co"
"machine5","ogit/MARS/Machine","Linux","machine 5","4G","contact@admin.co"
"machine6","ogit/MARS/Machine","Linux","machine 5A","8G","info@admin.co"
"machine7","ogit/MARS/Machine","Linux","machine 4A","4G","contact@admin.co"

Format of the CSV data for edges

Relevant REST endpoints:

POST https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/edges
DELETE https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/edges

The CSV has to have three headers:

  • from:ogit/_id or from:ogit/_xid

  • verb The ogit verb for the edge

  • to:ogit/_id or to:ogit/_xid

Example:

"from:ogit/_xid","verb","to:ogit/_xid"
"crew:NCC-1701-D:picard","ogit/Forum/mentions","crew:NCC-1701-D:data"

Format of the CSV data for timeseries

Relevant REST endpoints:

PUT https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/timeseries

The CSV has to have three headers:

  • ogit/_id or ogit/_xid

  • timestamp for ms since epoch or time for an ISO time

  • value for the value for this timestamp.

Example:

"ogit/_xid","timestamp","value"
"timeseries:machine4","1442235678000","45"
"timeseries:machine4","1442235944000","43"
"timeseries:machine5","1442235678000","51"
"timeseries:machine5","1442235944000","48"

or two headers:

  • ogit/_id or ogit/_xid

  • items for a JSON list of objects containing the keys (timestamp or time) and value.

Example:

"ogit/_xid","items"
"timeseries:machine4","[{""timestamp"": ""1442235678000"", ""value"":""45""},{""timestamp"": ""1442235944000"", ""value"":""51""}]"
"timeseries:machine5","[{""timestamp"": ""1442235678000"", ""value"":""43""},{""timestamp"": ""1442235944000"", ""value"":""48""}]"

Format of the CSV data for attachments

Relevant REST endpoints:

PUT https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/attachments

The CSV needs the following headers:

  • ogit/_id or ogit/_xid

One of the following headers is required for the content of the attachment:

  • content:text: Content to upload as simple text in UTF-8.

  • content:base64: Content to upload as base64 encoded binary data.

  • content:mimetype: (optional) Content-Type of the content.

Example:

"ogit/_xid","content:mimetype","content:text","content:base64"
"attachment:arago:test:0:lorem-ipsum","text/plain","Auch gibt es niemanden, der den Schmerz an sich liebt, sucht oder wünscht, nur, weil er Schmerz ist, es sei denn, es kommt zu zufälligen Umständen, in denen Mühen und Schmerz ihm große Freude bereiten können.

Um ein triviales Beispiel zu nehmen, wer von uns unterzieht sich je anstrengender körperlicher Betätigung, außer um Vorteile daraus zu ziehen? Aber wer hat irgend ein Recht, einen Menschen zu tadeln, der die Entscheidung trifft, eine Freude zu genießen, die keine unangenehmen Folgen hat, oder einen, der Schmerz vermeidet, welcher keine daraus resultierende Freude nach sich zieht? Auch gibt es niemanden, der den Schmerz ",""
"attachment:arago:test:1:lorem-ipsum","text/plain","","QXVjaCBnaWJ0IGVzIG5pZW1hbmRlbiwgZGVyIGRlbiBTY2htZXJ6IGFuIHNpY2ggbGllYnQsIHN1
Y2h0IG9kZXIgd8O8bnNjaHQsIG51ciwgd2VpbCBlciBTY2htZXJ6IGlzdCwgZXMgc2VpIGRlbm4s
IGVzIGtvbW10IHp1IHp1ZsOkbGxpZ2VuIFVtc3TDpG5kZW4sIGluIGRlbmVuIE3DvGhlbiB1bmQg
U2NobWVyeiBpaG0gZ3Jvw59lIEZyZXVkZSBiZXJlaXRlbiBrw7ZubmVuLgoKVW0gZWluIHRyaXZp
YWxlcyBCZWlzcGllbCB6dSBuZWhtZW4sIHdlciB2b24gdW5zIHVudGVyemllaHQgc2ljaCBqZSBh
bnN0cmVuZ2VuZGVyIGvDtnJwZXJsaWNoZXIgQmV0w6R0aWd1bmcsIGF1w59lciB1bSBWb3J0ZWls
ZSBkYXJhdXMgenUgemllaGVuPyBBYmVyIHdlciBoYXQgaXJnZW5kIGVpbiBSZWNodCwgZWluZW4g
TWVuc2NoZW4genUgdGFkZWxuLCBkZXIgZGllIEVudHNjaGVpZHVuZyB0cmlmZnQsIGVpbmUgRnJl
dWRlIHp1IGdlbmllw59lbiwgZGllIGtlaW5lIHVuYW5nZW5laG1lbiBGb2xnZW4gaGF0LCBvZGVy
IGVpbmVuLCBkZXIgU2NobWVyeiB2ZXJtZWlkZXQsIHdlbGNoZXIga2VpbmUgZGFyYXVzIHJlc3Vs
dGllcmVuZGUgRnJldWRlIG5hY2ggc2ljaCB6aWVodD8gQXVjaCBnaWJ0IGVzIG5pZW1hbmRlbiwg
ZGVyIGRlbiBTY2htZXJ6IAo="

Simultaneous handling of vertices, edges and/or attachments

Relevant REST endpoints:

PUT https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/combined

The format of the CSV data for vertices applies here, see above.

Edges

The format of an edge header is edge:[id]:[attribute]. [id] is an identifier to group edge commands together (most simple case: a number). This [id] must not contain a : since this is used as separator.

  • edge:[id]:verb: (required) Verb for that edge for the vertex of the current row.

  • edge:[id]:direction: ("in"/"out") from the view of the current vertex of that row. "in" points towards, "out" points away from the current vertex. Default is "out" if this column is missing.

One of the following headers is required to find the vertex to connect to:

  • edge:[id]:vertex_id: ogit/_id of the other vertex to connect to.

  • edge:[id]:vertex_xid: ogit/_xid of the other vertex to connect to.

Attachments

The format of an attachment header is content:[attribute]:

One of the following headers is required for the content of the attachment:

  • content:text: Content to upload as simple text in UTF-8.

  • content:base64: Content to upload as base64 encoded binary data.

  • content:mimetype: (optional) Content-Type of the content.

Example:

"ogit/_xid","ogit/_type","ogit/name","ogit/Forum/username","edge:0:verb","edge:0:direction","edge:0:vertex_xid","edge:1:verb","edge:1:direction","edge:1:vertex_xid"
"crew:NCC-1701-D:picard","ogit/Forum/Profile","Jean-Luc Picard","Picard","ogit/Forum/mentions","out","crew:NCC-1701-D:data","ogit/Forum/mentions","out","crew:NCC-1701-D:worf"
"crew:NCC-1701-D:worf","ogit/Forum/Profile","Worf","Worf","ogit/subscribes","out","crew:NCC-1701-D:picard","","",""
"crew:NCC-1701-D:data","ogit/Forum/Profile","Data","Data","ogit/Forum/mentions","out","crew:NCC-1701-D:worf","ogit/subscribes","in","crew:NCC-1701-D:worf"

Additional Documentation

A complete API documentation is available under https://csvconnector-881340721.eu-west-1.elb.amazonaws.com/openapi.json.

Use tools like Swagger or ReDOC to generate a readable documentation from this information. You could also use the Postman App, but this will only display JSON examples of the API.

arago GmbH ©2021 · Privacy Policy · Terms of Use · Imprint