Data Orchestration Ecosystem (DOE)
The Data Orchestration Ecosystem is for organisations needing to transport message data in real-time and near real-time in a controlled and managed approach between systems and other organisations. The DOE is an open-source, highly dynamic, event-driven workflow engine validating, enriching and transforming data to open standards in transit.
Features
- Vendor agnostic open-source platform
- Infrastructure that scales in line with your needs
- Event driven data transfer architecture
- Privacy by Design contracts to manage data flows
- Secure audit trails
- Dynamic work flow components
- Transform data to open standards between systems
- Reusable validation routines to ensure quality of data transferred
- Ingress & egress legacy file formats to consuming systems
- Built in resiliency and fault tolerance
Benefits
- No re-occurring license fees
- Extend and manage with internal resource
- Granular cost management of data and work flow processes
- Visualisation of data and work flow processes
- Effective use of development resource for data flow deployment
- Connect legacy data systems to modern end user applications
- Bring multiple data streams together into one interoperable compliant stream
- Manage data across organisational boundaries health social care and education
- Automatically scale infrastructure automatically in line with demand
- Supports "Green IT" through effective data centre energy management
Pricing
£5,000.00 to £50,000.00 an instance a month
- Education pricing available
Service documents
Request an accessible format
Framework
G-Cloud 14
Service ID
6 8 7 3 7 0 6 3 4 9 0 7 9 5 9
Contact
Hilltop Digital Lab
Gareth Roberts
Telephone: 07949049702
Email: gareth.roberts@hilltopdigitallab.com
Service scope
- Software add-on or extension
- No
- Cloud deployment model
-
- Public cloud
- Hybrid cloud
- Service constraints
- The Data Orchestration Ecosystem is an open source cloud based platform that utilises the market leading cloud providers and open source component service offerings including Kubernetes and Kafka.
- System requirements
-
- Access to cloud platform provider ie. Azure, AWS or GCP
- Internet Browser
User support
- Email or online ticketing support
- Email or online ticketing
- Support response times
-
As part of the support process, HD Labs provide a dedicated email address, telephone number and web portal to be used to raise support tickets.
During normal business hours (weekend by prior agreement) HD Labs will respond within 4 hours. - User can manage status and priority of support tickets
- Yes
- Online ticketing support accessibility
- WCAG 2.1 AAA
- Phone support
- Yes
- Phone support availability
- 9 to 5 (UK time), Monday to Friday
- Web chat support
- No
- Onsite support
- Yes, at extra cost
- Support levels
-
Standard support is covered in the cost. Service levels are based on priorities;
1) Critical Incident: eg Affecting all DOE users. Response 1 hour & Resolution 8 hours
2) Major incident: eg Component/route failure for users.Response 2 hours & Resolution 2 days
3) Minor Incident: eg performance impacted for some users. Response 4 hours & Resolution 5 days
4) Request for change: eg, new functionality requested. Response 2 days & Resolution Planning 28 days and is based on a maximum
Support calls are routed to 2nd line support as appropriate.
Customers have access to a named account/project manager.
On-site support is available by arrangement and is charged on a time and materials basis (see Rate Card). - Support available to third parties
- Yes
Onboarding and offboarding
- Getting started
-
HD Labs follow a multi-phase approach to the onboarding process, which provides a robust, agile and rapid approach to capability provision that has been tested, refined and proven. The phased approach and sequence follow the Government Digital Service i) Discovery ii) Alpha iii) Beta iv) Live.
Workshops are an integral part of the discovery process to help teams understand the flow, the use of the data and the end-users using the data in a day to day scenario. Technical users are trained in the setup of the data contracts and the workflow sequencing which is a drag-and-drop interface utilising components from an on-screen toolbox which are dropped onto the canvas and connected together to form a flow which gets translated into the infrastructure. User documentation is supplied through a wiki for technical users to understand the underlying logic and architecture and API interfaces are described through swagger / openAPI. Environments are self-documenting and therefore technical teams always can see the set-up and configuration at any given time. - Service documentation
- Yes
- Documentation formats
-
- HTML
- Other
- Other documentation formats
- Wiki
- End-of-contract data extraction
- HD Labs can export all existing data in its platform into raw formats. The DOE is an Open Source Platform built on open standards and designed to prevent vendor lock-in. All data in the platform can be securely exported in non-proprietary formats including open source databases for use in other systems or databases. HD Labs will work with the customer to determine the best export format for their destination systems.
- End-of-contract process
-
The DOE is an open-source application built using an open-source technology stack removing platform license costs and allowing local development teams to build and extend the platform as required.
At the end of the contract, it is expected that HD Labs will be able to hand over the ongoing maintenance to local development teams to continue development.
As part of that handover is the source code including components and and infrastructure (IaC)
Using the service
- Web browser interface
- Yes
- Supported browsers
-
- Internet Explorer 11
- Microsoft Edge
- Firefox
- Chrome
- Safari
- Opera
- Application to install
- No
- Designed for use on mobile devices
- No
- Service interface
- Yes
- User support accessibility
- None or don’t know
- Description of service interface
- The visual front end management interface provides the ability to build and manage the data contracts between the source and consuming systems. The visual interface provides authorised users such as Information Governance leads to approve or reject data contracts. Once a contract is approved it is submitted to the Kubernetes control plane and a controller will interpret this contract and convert it into the resources needed to support the data contract including any roles, service accounts, pods, services, configurations, secrets, Kafka topics, load balancers, DNS and subdomains as required.
- Accessibility standards
- None or don’t know
- Description of accessibility
-
The service interface is a management console allowing users to set up and manage data contracts and the workflow that occurs as data transits the DOE.
The interface is a ReactJS implementation using Material UI and allows users to drag and drop components onto the canvas and sequence them by drawing lines between components.
Selecting components displays the properties associated with that component. Once a contract has been designed it is submitted to an approver role to be approved.
The design follows the principles of being clear, robust and specific. - Accessibility testing
-
Testing has been conducted manually using tools such as Selenium accross a number of different browser types.
Accessibility features such as "dark mode" are enabled and the interface is compatible with browser and windows magnifers. - API
- Yes
- What users can and can't do using the API
-
Users can create a variety of dynamic data flows and API gateways. These APIs allow customers to send data onto or receive data from one or many other systems as defined in the integration contracts.
The DOE Supports the following interfaces and standards;
- SOAP
- REST
- RPC
Standards supported include: FHIR, JSON, XML, HL7, IHE, DICOM, OAUTH, Openehr etc
APIs are available to stream audit data to a trusted audit aggregator.
Users cannot use the APIs to run services which are not in the component registry. - API documentation
- Yes
- API documentation formats
-
- Open API (also known as Swagger)
- Other
- API sandbox or test environment
- Yes
- Customisation available
- Yes
- Description of customisation
-
Users can create a variety of dynamic data flows and API gateways. These APIs allow customers to send data onto or received data from one or many other systems as defined in the data contracts.
The works flows are customisable through the visual Management interface and components for validation, transformation and enrichment can be added and the route directed to a destination of choice.
The DOE is built on open-source architecture therefore components such as new validation routines, data enrichment sources and transformation can be generated without proprietary knowledge.
Additionally, the way the architecture is built is a modular micro services approach orchestrated by Kubernetes where the components that are required are micro and do not require complicated call and service setup as this is handled by Kubernetes. Data routes can be customised further by defining aspects such as cost centres for resource usage and thresholds by scaling resources up or down in response to changes in demand. The open-source architecture of the DOE allows for extensive customisation.
Scaling
- Independence of resources
-
Performance and scaling is central to the core design of the DOE. The DOE is centred around the setup of Kubernetes pods and the use of Microservices to ensure that data routes are separated and operate independently from one another. This means that routes consuming high volumes of activity scale in line with demand and do not affect other routes using moderate demand.
Once the infrastructure is provisioned it is monitored by another controller service to manage demand and also to scale back in when demand is lower.
Analytics
- Service usage metrics
- Yes
- Metrics types
-
Metrics are provided for
i) Each Data route and include cost per route, storage used, compute time, number of transactions, transaction size
ii) Support provided including response and resolution against the SLA, User requests and tickets per data route comparison
iii) Ecosystem overall including cost, compute time, uptime, transaction numbers and storage - Reporting types
-
- API access
- Real-time dashboards
- Regular reports
- Reports on request
Resellers
- Supplier type
- Not a reseller
Staff security
- Staff security clearance
- Conforms to BS7858:2019
- Government security clearance
- Up to Developed Vetting (DV)
Asset protection
- Knowledge of data storage and processing locations
- Yes
- Data storage and processing locations
- United Kingdom
- User control over data storage and processing locations
- Yes
- Datacentre security standards
- Managed by a third party
- Penetration testing frequency
- At least once a year
- Penetration testing approach
- Another external penetration testing organisation
- Protecting data at rest
-
- Physical access control, complying with CSA CCM v3.0
- Physical access control, complying with SSAE-16 / ISAE 3402
- Data sanitisation process
- No
- Equipment disposal approach
- A third-party destruction service
Data importing and exporting
- Data export approach
-
A rich set of APIs enable data to be accessed as it transits the DOE and is validated, Enriched and transformed via the workflows described in the Data contracts. As a result, the data can be exported in any number of formats appropriate to the destination systems and their capabilities including topic-based queues, databases, or messages conforming to open standards such as Fhir, HL7, IHE or to openEHR.
Data contracts are set up to enable data to be passed in a managed way.
HD Labs will work with the customer to determine the best export format for their destination systems. - Data export formats
-
- CSV
- Other
- Other data export formats
-
- Open Source Database
- FHIR
- TXT
- JSON
- XML
- HL7
- Data import formats
-
- CSV
- Other
- Other data import formats
-
- FHiR
- XML
- JSON
- DICOM
- TXT
- CDA
- XLSX
- Doc
- JPG
Data-in-transit protection
- Data protection between buyer and supplier networks
-
- Private network or public sector network
- TLS (version 1.2 or above)
- IPsec or TLS VPN gateway
- Data protection within supplier network
-
- TLS (version 1.2 or above)
- IPsec or TLS VPN gateway
Availability and resilience
- Guaranteed availability
- Level of availability varies depending on the specific project and requirements.
- Approach to resilience
-
The underlying cloud infrastructure is highly available across multiple availability zones. Each zone is designed to eliminate single points of failure (like power, network & hardware). As example by utilising the three availability zones in the AWS London region we are able to protect against a disaster at any of the datacenters.
Within AWS implementations; The Amazon Elastic Container Service for Kubernetes (EKS) cluster is distributed across the three availability zones and the workload is spread across the nodes using logic inside of the kubernetes service. In the event of an outage, the workload will be redistributed across the remaining nodes and the autoscale configuration allows the EKS clusters on the remaining sites to scale outwardly to cope with the reduction in service. The Amazon Managed Streaming for Apache Kafka (MSK) cluster is distributed across the three availability zones and the queues partitions are automatically replicated to the other zones to ensure no data loss. The Amazon Elastic MapReduce (EMR) service is distributed across the three availability zones and the workload is spread across the nodes using the MapReduce functionality.
In addition all infrastrcuture and and workflow processsess are as code therefore they are designed to be repeatable and consistant. - Outage reporting
-
In addition to the monitoring tools in place the DOE can email pre-defined customer and support groups.
We can potentially also automatically raise tickets on a variety of support desks that have API integration capabilities .
Identity and authentication
- User authentication needed
- Yes
- User authentication
-
- 2-factor authentication
- Public key authentication (including by TLS client certificate)
- Identity federation with existing provider (for example Google Apps)
- Limited access network (for example PSN)
- Dedicated link (for example VPN)
- Username or password
- Other
- Other user authentication
-
An authentication provider will provide functionality for the system to login to other applications and to authenticate other applications when they are making calls to the DOE.
The DOE defines scopes for permissions, these scopes will be mapped by the authentication provider to state what access an individual or application has when they are logged into our platform.
The authentication provider will provide policy based access controls for any users or applications accessing the system. These roles should limit access to only what is needed based on a variety of attributes which will be assessed during the login process. - Access restrictions in management interfaces and support channels
- HD Labs solutions are cloud-agnostic. However, as an example, 2 factor authentication is required on the AWS Management Console to manage the AWS account. For Infrastructure changes, 2-factor authentication across a VPN connection is required. VPN is established using public-key authentication. Access is a restricted Roles based access control to the management interface that allows users to set up and approve/reject electronic data contracts. Similarly, the Service support channel via a dedicated web portal requires a username and password in order to log in and raise a support ticket and view the progress of an existing ticket.
- Access restriction testing frequency
- At least every 6 months
- Management access authentication
-
- 2-factor authentication
- Public key authentication (including by TLS client certificate)
- Identity federation with existing provider (for example Google Apps)
- Limited access network (for example PSN)
- Dedicated link (for example VPN)
- Username or password
- Other
- Description of management access authentication
- We provide OAUTH2 services via keycloak, this can plug into a variety of other OAUTH2 services such as CIS2, NHS futures, as well as providing capabilities of providing authentication through other methods such as SAML. This can be extended to mapping our roles to roles as defined over on the source system which ensures that we can integrate with many of our customers SSO offerings and allow them to manage user access through their own roles-based access controls.
Audit information for users
- Access to user activity audit information
- Users have access to real-time audit information
- How long user audit data is stored for
- User-defined
- Access to supplier activity audit information
- Users have access to real-time audit information
- How long supplier audit data is stored for
- User-defined
- How long system logs are stored for
- User-defined
Standards and certifications
- ISO/IEC 27001 certification
- No
- ISO 28000:2007 certification
- No
- CSA STAR certification
- No
- PCI certification
- No
- Cyber essentials
- Yes
- Cyber essentials plus
- Yes
- Other security certifications
- No
Security governance
- Named board-level person responsible for service security
- Yes
- Security governance certified
- No
- Security governance approach
- All systems and components utilise end-to-end encryption. All data is encrypted at rest. Audits records are signed in a chain to ensure a tamper evident solution. Audits can be exported to customers own audit analysis and reporting solution. Each integration contract includes a description of what that integration is doing, what it is for and provides metadata allowing the contract to be linked to data sharing agreements. Each contract can be reviewed and approved by IG officer using the management portal before processing is allowed.
- Information security policies and processes
-
We are currently working to attain ISO27001 accreditation, risk registers exist for each project and are updated regularly. Change processes exist to ensure that customers are made aware of any changes. Components always use protocols and services which ensure that data is encrypted both in transit and at rest.
Developments are thoroughly tested prior to release with test plans updated to reflect new and emerging threats and vulnerabilities this is built into the deployment pipeline as releases move from Development environments to Quality Assurance environments before progressing to User Acceptance and production environments.
HD Labs also have the following certifications
- Certified Cyber Essentials Plus, registration IASME-CEP-007590.
- Registered with the Information Commissioners Office (ICO), registration ZA794790
- Registered with NHS Data Security Protection Toolkit (DSPT), organisation code 8KL37
HD Labs also have executive board nominations for Caldicott Guardian; Senior Information Risk Officer (SIRO); Information Governance Lead and Data Protection Officer (DPO).
Operational security
- Configuration and change management standard
- Supplier-defined controls
- Configuration and change management approach
-
HD Labs will work with the organisation in advance of any changes and jointly assess:
- the change itself.
- the risk of not implementing the change.
- the back out plans of any change to ensure that the service or system can be rolled back (restored) to its pre-change status.
HD Labs utilise an agile product development process and visibility of upcoming changes are visible to the wider service teams well in advance of release schedules.
All Configuration and Changes are sorce controlled including IaC - Vulnerability management type
- Supplier-defined controls
- Vulnerability management approach
-
We utilise automated documentation tools to assist in the documentation of software used by our solution, this includes container images and pre-requisites to prevent differences between environments.
The deployment pipeline includes automated test scripts to check for vulnerabilities. The CTO and Product Delivery roles also subscribe to sites including the National Cyber Security Center (NCSC), OWASP and NHS digital
Vulnerabilities are raised on the project customer risk registers. We investigate the severity of the vulnerability on our software and again update the risk register with advice and mitigation plans and ensure scheduled remediation. - Protective monitoring type
- Supplier-defined controls
- Protective monitoring approach
-
The infrastructure will be continuously monitored using a number of tools specifically focusing on cluster and queue performance and associated actions when communicating between the source and destination systems, and workflow actions.
Infrastructure is monitored in terms of capacity and that as the Organisations dataflow expands and therefore the volume of referral traffic increases that the infrastructure is monitored to ensure appropriate capacity. - Incident management type
- Supplier-defined controls
- Incident management approach
-
The Technical Service agents are responsible for logging each initial contact as a “case” on the Jira Service Management application, they also provide an escalation to other team members in Development and Infrastructure.
Technical Service Agents are responsible for managing the “case” through to resolution with the customer and where necessary will call on expertise in other teams to help with resolution as well as escalate cases to managers as appropriate. We will refer the call to the relevant support desks where appropriate such as supplying or consuming data systems
Secure development
- Approach to secure software development best practice
- Conforms to a recognised standard, but self-assessed
Public sector networks
- Connection to public sector networks
- Yes
- Connected networks
-
- NHS Network (N3)
- Health and Social Care Network (HSCN)
Social Value
- Social Value
-
Social Value
- Tackling economic inequality
- Equal opportunity
Tackling economic inequality
Data orchestration plays a crucial role in supporting the delivery of the UK government's Procurement Policy Note (PPN) 06/20, specifically in the social value theme of tackling economic inequality, through
i) Creation of training opportunities for IT / digital personnel in public sector organisations.
ii) Support of innovation and disruptive technologies through the provision of healthcare data to deliver efficiencies and /or higher quality services.
iii) Support the development of scalable and future-proofed new methods to modernise delivery and increase productivityEqual opportunity
Data orchestration plays a crucial role in supporting the delivery of the UK government's Procurement Policy Note (PPN) 06/20, specifically in the social value theme of Economic opportunity, through;
i) Support in-work progression to help people, including those from
disadvantaged or minority groups, to move into higher-paid work by
developing new skills with the public sector organisations.
ii) Enable data flow from specialist patient services into healthcare ecosystem
Pricing
- Price
- £5,000.00 to £50,000.00 an instance a month
- Discount for educational organisations
- Yes
- Free trial available
- No