• First Steps
  • Basic Data Operations
  • Creating Action Handlers using Action API
  • Developing a Connector
  • Working with Automation Tasks

Overview

In this tutorial, we will show how to write a simple connector for HIRO. We have picked Google Calendar as example application. The connector we develop in this tutorial will read events from a calendar and copy them to the HIRO knowledge core. To keep the connector simple, we will copy the events only in one direction. Synchronization of data will be covered by another tutorial.

The code examples are written in Python, but the HIRO API can be used with any other programming language, as well.

google connector architecture
Figure 1. Your connector integrates HIRO with Google Calendar

We will perform the following steps to implement the connector:

  1. Prepare prerequisites in Google Calendar / Google Cloud

  2. Connect with HIRO

  3. Fetch events from Google Calendar

  4. Write retrieved events to HIRO

  5. Synchronization: only write new or updated events

  6. Create a loop to continuously poll events

Prerequisites & Preparations

Preparing a Google service account

We want to use the Google API to read data from a Google calendar. In order to do this, we need user and application credentials, similiar to the HIRO API. The required steps are: . Create a google cloud account . Create a new project . Create Service Account credentials new app/project

If you already have the required accounts and credentials, you may skip to the next section. Otherwise, continue to read the step-by-step instructions.

Create a google cloud account

Go to https://cloud.google.com/ and sign in. If you don’t have a Google Cloud account, you have to register first.

Create a new project

In the Google Cloud Platform, users, credentials, and permissions are organized in projects. We will create a new project "HIRO-Connector" in the Google Cloud.

Proceed to the Google Cloud console: https://console.cloud.google.com/ , and click on the project drop-down box on top. Click on "NEW PROJECT" to create a new project. Now enter the name of your new project: "HIRO-Connector".

GCP project selection
Figure 2. Create new GCP project
GCP new project
Figure 3. Enter project name

You have created a new Google Cloud project. As next steps, we will create an API key for our application, and a service account.

Create Service Account credentials new app/project

From the Google Cloud Console home screen, select "Credentials" from the "API & Services menu". Click on "create credentials", and then "Create service account".

GCP create credentials 1
Figure 4. GCP - APIs & Services
GCP create credentials 2
Figure 5. GCP - New credentials

We enter a new service account name, which we call "hiro-connector" and press create. On the next screen select the role as editor and click continue. On the last screen click done.

GCP create credentials 3
Figure 6. GCP - Create service account
GCP create credentials 3 1
Figure 7. GCP - Edit role

On the service account page, click on the button next to key creation date and press create key.

GCP create credentials 3 2
Figure 8. GCP - Create private key

The key type should be JSON. Press the "Create" button, and you’ll be prompted to save the credentials file to disk. This file contains all necessary credentials data to connect to the API.

GCP create credentials 4
Figure 9. GCP - Download private key

To grant calendar permissions to this service account, you will need the account’s email address later. Look it up in the GCP console while you are still there. It should look like this: hiro-connector@hiro-connector-123456.iam.gserviceaccount.com

Preparing a Google Calendar

We will now create a new Google calendar to work with. You may of course also work with an existing calendar, if you prefer.

Create a new Google calendar Go to Google Calendar (https://calendar.google.com/) and create a new calendar ("Other calendars" -→ "+" -→ "Create new calendar". Enter "HIRO-TEST" as name.

new google calendar
Figure 10. New Google calendar

Next. we enter the calendar setting for the "HIRO-TEST" calendar. We share our calendar with the service account hiro-connector@hiro-connector-123456.iam.gserviceaccount.com and provide it edit permissions.

share google calendar
Figure 11. Share Google calendar

Copy the calendar ID from the settings page - you will need this later to access the calendar through the API.

Enable Calendar API for the new project (in Google Developer Console)

There is one last step that we have to perform: enable the Calendar API for your new project. We do this in the Google Developer Console: https://console.developers.google.com/

Select the "HIRO Connector" project, and click on "+ ENABLE APIS AND SERVICES". Select "Google Calendar API", and enable it on the next screen. Your are ready now to connect via API to your calendar.

enable calendar api
Figure 12. Enable calendar API

Install Python libraries for Google API

You will need two Python libraries for the Google API. In case you do not have them in your environment yet, install them with the pip command.

Google Oauth client:

pip install google-auth

Google Python client library:

pip install google-api-python-client

Preparing HIRO session

Finally, we initiate a HIRO session, as seen in the First Steps with HIRO tutorial:

# Initiate session
hiro_session = initiate_hiro_session('./hiro_credentials.json')

We will also need the functions create_vertex, update_vertex and create_edge, which we have defined in the Basic Vertex Operations tutorial.

Connecting with the Google API

Before we can read and write data via the Google API, we need to authenticate and create a session with the intended usage scope. We load the service account credentials, which we had downloaded from the Google Cloud Platform Console before (see above), set the scope to 'https://www.googleapis.com/auth/calendar', and create an authenticated session.

# google api imports
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
from googleapiclient.discovery import build

# configuration
google_credentials_file = "./forward-aura-168008-92a0f1f9cd3a.json"         #replace with your credentials file!
google_calendar_id = '7hfea2godbmajrspfp30l23q1o@group.calendar.google.com' #replace with the id of your calendar!

# load credentials and authorize session
credentials = service_account.Credentials.from_service_account_file(google_credentials_file)
scoped_credentials = credentials.with_scopes(['https://www.googleapis.com/auth/calendar'])
authed_session = AuthorizedSession(scoped_credentials)

Once we are authenticated and have created a session with the Google API, we can instantiate a specific service for the Calendar API:

# Build Calendar API Service
service = build('calendar', 'v3', credentials=scoped_credentials)

Reading calendar data from Google

As next step, we will check that we can read calendar data through the Google API, to verify that we got everything in place correctly. As a test, we print out the calendar name to the console, and retrieve all calendar events.

# get calendar name
calendar = service.calendars().get(calendarId=google_calendar_id).execute()
print ('Calendar name: ' + calendar['summary']+ '\n')

print('Getting all events')
events_result = service.events().list(calendarId=google_calendar_id, singleEvents=True, orderBy='startTime').execute()
events = events_result.get('items', [])

if not events:
    print('No upcoming events found.')
for event in events:
    start = event['start'].get('dateTime', event['start'].get('date'))
    print(start, event['summary'])

We have read data from Google Calendar and printed it out to the console. We are ready for the final step: writing the calendar data to HIRO.

Writing calendar data to HIRO

For writing calendar data to HIRO, we define two helper functions. The first function write_calendar() creates a node in the HIRO Graph, which represents the specific calendar. We add the calendar id as _xid and calendar name as title attribute, so that we know which calendar it is.

The second helper function write_event() creates a vertex of type ogit/Schedule/Event in HIRO for the calendar event from Google. In this example, we write the event id, title, start and end time, description, location, and etag as attributes. You might want to extend this with additional attributes in the future. The new event vertex is connected with a ogit:contains edge to the calendar vertex.

def create_calendar (calendar):
    vertex_data = {'ogit/title':calendar['summary'],
                   'ogit/_xid': 'CAL-' + calendar['id'],
                   '/etag': calendar['etag']
                  }

    return create_vertex('ogit/Schedule/Calendar', vertex_data)

def create_event (cal_id, event):

    vertex_data = {'ogit/title':event['summary'],
                   'ogit/Schedule/startTime': event['start'].get('dateTime', event['start'].get('date', '')),
                   'ogit/Schedule/endTime': event['end'].get('dateTime', event['end'].get('date','')),
                   'ogit/_xid': 'CAL-' + cal_id + '-' + event['id'],
                   '/etag': event['etag']
                  }

    if 'description' in event:
        vertex_data['ogit/description'] = event['description']
    if 'location' in event:
        vertex_data['ogit/Schedule/location'] = event['location']

    vertex_id = create_vertex('ogit/Schedule/Event', vertex_data)
    create_edge(cal_id, vertex_id, 'ogit:contains')

    return vertex_id

Now we can retrieve the calendar data from Google again, and write the entries to HIRO. We will first create a calendar item in HIRO. Afterwards, we iterate over all calendar events, copy each event to HIRO, and connect it to the calendar vertex.

# get calendar name
calendar = service.calendars().get(calendarId=google_calendar_id).execute()

# create calendar vertex in HIRO
hiro_cal_id = create_calendar(calendar)

# retrieve all events from Google calendar
events_result = service.events().list(calendarId=google_calendar_id, singleEvents=True, orderBy='startTime').execute()
events = events_result.get('items', [])

for event in events:
    hiro_event_id = create_event (hiro_cal_id, event)

If you check now, you will find your calendar and all events in the HIRO Graph.

Continuous synchronization: repeatedly write new or updated events

Of course, we don’t want to create new data objects every time we check the Google calendar, but we want to update the data in HIRO when there are changes or new events in the Google calendar. Therefore, we need to first check whether there are already nodes in HIRO for the calendar or event; and if they exist, we want to know whether there have been any changes that need to be updated.

HIRO offers a special 'ogit/_xid' attribute, which should be used to specify the external id that the item has in the original data source. If you write for instance the Google event id to this field, you will be able to check whether a specific event is already stored in HIRO or not. There is also a convenience method in the Graph API that facilitates looking up vetices by external id.

External ids are supposed to be unique in the HIRO database. Therefore, we do not just use the calendar or event ids in our example, but add a prefix, so that we reduce the probability of id conflicts. This is a good practice that you might want to adopt already in your first connector.

Google has introduced a nice feature for determining whether a data object has changed: there is an etag attribute, which is assigned a new random value when the object is changed. As we have already copied this attribute to the HIRO vertices before, we can simply check whether the etag has changed. Thanks Google!

We define helper methods to check whether the calendar and events already exist and whether there are changes:

def update_calendar (vertex_id, calendar):
    vertex_data = {'ogit/title': calendar['summary'],
                   'ogit/description': calendar['description'],
                   '/etag': calendar['etag']
                  }

    return update_vertex(vertex_id, vertex_data)

def read_vertex_by_xid (xid):
    response = hiro_session['session'].get(hiro_session['graph_api'] + 'xid/' + xid )

    if response.ok:
        if len(response.json()['items'])==1:
           return response.json()['items'][0]
    else:
        print (str(response.status_code) + " - " + response.text)
        print ('Failed to read vertex.')
    return None

def create_or_update_calendar (calendar):
    xid = 'CAL-' + calendar['id']
    etag = calendar['etag']

    cal_vertex =  read_vertex_by_xid (xid)

    if cal_vertex == None:
        return create_calendar (calendar)
    else:
        print (cal_vertex)
        hiro_etag = cal_vertex.get('/etag', '')
        vertex_id = cal_vertex ['ogit/_id']
        if etag != hiro_etag:
            update_calendar (vertex_id, calendar)
        return vertex_id


def update_event (vertex_id, cal_id, event):
    vertex_data = {'ogit/title':event['summary'],
                   'ogit/Schedule/startTime': event['start'].get('dateTime', event['start'].get('date', '')),
                   'ogit/Schedule/endTime': event['end'].get('dateTime', event['end'].get('date','')),
                   '/etag': event['etag']
                  }

    if 'description' in event:
        vertex_data['ogit/description'] = event['description']
    if 'location' in event:
        vertex_data['ogit/Schedule/location'] = event['location']

    return update_vertex(vertex_id, vertex_data)


def create_or_update_event (cal_id, event):
    xid = 'CAL-' + cal_id + '-' + event['id']
    etag = event['etag']

    event_vertex =  read_vertex_by_xid (xid)

    if event_vertex == None:
        create_event (cal_id, event)
    else:
        hiro_etag = event_vertex.get('/etag', '')
        vertex_id = event_vertex ['ogit/_id']
        if etag != hiro_etag:
            update_event (vertex_id, cal_id, event)

And now we can adapt our synchronization routine to utilize the new methods:

import time

# get calendar name
calendar = service.calendars().get(calendarId=google_calendar_id).execute()

# create calendar vertex in HIRO
hiro_cal_id = create_or_update_calendar(calendar)

#endless loop
while True:
    # retrieve all events from Google calendar
    events_result = service.events().list(calendarId=google_calendar_id, singleEvents=True, orderBy='startTime').execute()
    events = events_result.get('items', [])
    for event in events:
        hiro_event_id = create_or_update_event (hiro_cal_id, event)

    time.sleep(60)

Try it out by adding new events and changing events in your Google calendar. Within the endless loop, the connector code checks once per minute for new or changed data. Only new or changed events will be written to the HIRO Graph.

Calendar event in HIRO Graph
{
  "/etag" : "3130664879208000",
  "ogit/Schedule/endTime" : "2019-08-09T10:30:00+02:00",
  "ogit/Schedule/location" : "somewhere else",
  "ogit/Schedule/startTime" : "2019-08-09T10:00:00+02:00",
  "ogit/_created-on" : "Fri, 09 Aug 2019 06:31:28 GMT",
  "ogit/_creator" : "user1@academy.local",
  "ogit/_creator-app" : "cjix82rxi000gu473w5kvkpqv",
  "ogit/_graphtype" : "vertex",
  "ogit/_id" : "cjz3qffjoci6bgp027kuzwx5g",
  "ogit/_is-deleted" : "false",
  "ogit/_modified-by" : "user1@academy.local",
  "ogit/_modified-by-app" : "cjix82rxi000gu473w5kvkpqv",
  "ogit/_modified-on" : "Fri, 09 Aug 2019 06:34:30 GMT",
  "ogit/_owner" : "academy.local",
  "ogit/_type" : "ogit/Schedule/Event",
  "ogit/_v" : "3",
  "ogit/_v-id" : "1565332470438-Swxvdw",
  "ogit/_xid" : "CAL-cjz1frals2bn2gp02iq46j9ex,-57f7qq54o0hf25sr3jcereti4b",
  "ogit/title" : "TEST"
}

Summary

In this tutorial, we have implemented our first HIRO connector.The connector imports calendar events from a Google calendar into the HIRO Graph.

We have performed the following steps for implementing this connector:

  • Created a new project and a service account in Google Cloud Platform

  • Created a new Google Calendar and configured permissions

  • Established an authenticated session with HIRO

  • Fetched events from Google Calendar

  • Written retrieved calendar events to HIRO

  • Create a loop to continuously poll events and write changes to HIRO

As you probably have noticed, we have copied only a subset of the calendar event attributes to HIRO. You may extend the code to make more attributes available.

Now that you have calendar data in HIRO, you can automate some tasks to manage your calendar. If you haven’t read the tutorial about creating automation tasks yet, we suggest to continue there.