Retrieving Data
The first step performed when a policy is evaluated is to retrieve the data that needs to be validated. There are two ways to retrieve data from a policy template:
| • | Resources make it possible to list resources using the Flexera APIs. |
| • | Datasources make it possible to retrieve data from any arbitrary service. |
Datasources can also be used to further process data retrieved from either resources or other datasources as described in Chaining Datasources.
The below links provide resources for retrieving data:
| • | Listing Resources |
| • | Including Tags |
| • | API Data With Datasources |
| • | Authorization |
| • | Pagination |
| • | Request |
| • | Result |
The resources definition identifies a resource type and an optional set of filters used to list resources:
resources <name>, type: <resource type> do
filter do
<filter> <filter comparator> <filter value>
<filter> <filter comparator> <filter value>
...
end
<property> <property value>
<property> <property value>
...
end
| • | name defines the resource name and must be unique. It can be referenced using @ elsewhere in the code. |
| • | resource type is the type of resource. |
| • | filter is the API field to filter on, such as state. If multiple filters are specified it will AND together the results. Only resources which satisfy all the filters will be returned. |
| • | filter comparator is optional and must be either ne: or eq:. If not specified, it defaults to eq:. |
| • | filter value is either a term or array of terms. |
Important:Only ever specify an array of terms with ne: as the comparator. If you specify it with eq: it will do a logical AND and try and set the filter be equal to ALL the values passed in, which will return no results.
| • | property is a property of the resource such as cloud_href or view. |
| • | property value is a value of the property. |
This example filters returns active Windows instances on AWS clouds. The first resources block filters upon cloud_type to only return AWS regions. The filter comparator is left off and defaults to eq:. The second resources block filters upon the os_platform and state of the instances to get active instances. It can be referenced later in the policy as @instances.
resources "aws_regions", type: "rs_cm.clouds" do
filter do
cloud_type "amazon"
end
end
resources "instances", type: "rs_cm.instances" do
iterate @aws_regions
cloud_href href(iter_item)
filter do
os_platform "windows"
state ne: ["stopped", "provisioned", "terminated"]
end
end
You can query for resources by tag using tags field in your resource. You can also use the all, any, none methods to expand your search with tags. Only one tag field is supported per resource.
resources "instances", type: "rs_cm.instances" do
iterate @aws_regions
cloud_href href(iter_item)
tags "ns:name=*"
end
resources "volumes", type: "rs_cm.volumes" do
iterate @aws_regions
cloud_href href(iter_item)
tags any(["ns:name=*","foo:bar=hurray"])
end
# include fields in your datasource
datasource "instances" do
iterate @instances
field "href", href(iter_item)
field "id", val(iter_item,'resource_uid')
field "name", val(iter_item,'name')
field "state", val(iter_item,'state')
field "type", "instances"
field "tags", val(iter_item,'tags')
end
Additional tag filters examples.
tags "ns:name=*" # Filter by namespace and key
tags "ns:*"# Filter by namespace only
tags "*"# Return any resource with a tag
tags any(["ns:name=*","foo:bar=hurray"]) # Resource must include any tag in array.
tags all(["ns:name=*","foo:bar=hurray"]) # Resource must include all tags in array.
tags none(["ns:name=*","foo:bar=hurray"]) # Resource must not include any tags in array
There are three forms of datasource definitions. The first one, discussed here makes it possible to retrieve data from HTTP APIs. The second one, described in Processing Datasources With JavaScript makes it possible to use JavaScript to process existing data and the last one described in Chaining Datasources makes it possible to derive a datasource from existing data.
Authorization against any external APIs that a policy uses is handled using credentials entered in the Automation credentials page (see Managing Credentials for Policy Access to External Systems). To use credentials to make API calls, add a credentials declaration in the policy template. This declaration specifies all the details needed to use credentials and will allow the user of the policy to select the appropriate credential when applying the policy.
credentials <name> do
schemes <type1>, <type2>
label <label>
description <description>
tags <tag filters>...
aws_account_number <parameter reference>
end
| • | name is the name in the policy template language. The credential is referenceable by this name in datasource and define declarations. |
| • | schemes is the authorization scheme in the credentials service and must be one of aws_sts, aws, basic, ntlm, api_key, jwt, or oauth2, matching the API that the credential is used with. Multiple schemes can be listed if the credential and code work with multiple types. When a user selects a credential on the Applying Policies screen, they will only be able to select credentials that are of one of the specified schemes. |
| • | label is a short human readable label for the credential -- it is shown to the user on the Applying Policies screen |
| • | description is a longer description of what the credential is used for in the policy and is also shown to the user on the Applying Policies screen |
| • | tags is an optional field used to filter the credentials that are shown to a user when applying a policy. It may contain tags in the form of key=value. By default, the credential management UI and all Flexera-built policies use the tag key provider for credential matching purposes. |
| • | aws_account_number is an optional field used by aws_sts credentials to allow for a single credential to be used on multiple accounts. It must be passed as a parameter with a type of string. If aws_account_number is to be optional during policy application, the parameter must have a default value of an empty string. aws_sts must be a listed scheme to use aws_account_number. If an aws_sts credential is not used during policy application, the passed value will not be considered during policy evaluation, but will still be required unless a default value for the associated parameter was set. |
Each credentials declaration will require a separate selection from the user.
Note:Older Policies may use define authorization inline using the auth declaration, but this approach is no longer considered best practice.
The following example will authenticate against an AWS API using either an aws or aws_sts scheme type and export a $cred_aws reference.
credentials "cred_aws" do
schemes "aws", "aws_sts"
label "AWS Credential"
description "This credential should have read/write access to AWS EC2 Instances"
tags "provider=aws"
aws_account_number $param_aws_account_number
end
The following example will authenticate against an Azure API using either an oauth2 scheme type. Since there many be many APIs using the oauth2 scheme, an optional tags declaration is added to this example. Credentials can be tagged with arbitrary key value pairs. In this case, this block is telling it to filter for credentials marked with cloud=azure”
credentials "cred_azure_compute" do
schemes "oauth2",
label "Azure compute credential"
description "Enter a credential with read/write access to Microsoft.Compute resources."
tags "provider=azure"
end
It is highly recommended to provide a provider tag filter and then set the provider field when entering credentials into the dashboard so policy managers can easily find credentials when applying the authored policy.
If results from a custom datasource are paginated, you must define a pagination block above your datasource that describes how to extract the next page token from a response and where to insert the token into subsequent queries.
pagination <name> do
get_page_marker do
body_path <path term>
header <header name>
end
set_page_marker do
query <query param name>
header <header name>
body_field <body field name>
uri <true|false>
end
| • | <name> is the name of the paginator. The name can be referred to in the pagination field of datasource request blocks with a $ in front. |
| • | <path term> is a call to either the jmes_path, jq, or xpath method describing how to extract the page token from the response body (for backward compatibility, it can also be a string which will be interpreted as either JMESPath for JSON encoding or XPath for XML encoding). |
| • | <header name> is the name of the http header containing the page token. |
| • | <query param name> is the name of the query string parameter to set. |
| • | <body field name> is the name of the field in the body to set. |
| • | uri should be set to true if the page token marker is a full URL, such as a nextLink attribute returned by Azure ARM services. |
For get_page_marker exactly one of body_path or header must be set. For set_page_marker exactly one of query, header, body_field, or uri must be set.
Many AWS API requests contain a variable in the response body called PageToken. This variable should be set as a HTTP query parameter NextToken to subsequent requests. AWS supports both XML and JSON encoding. The datasource definition has an encoding of xml below so an XPath expression is used for the body_path.
pagination "aws_pagination_xml" do
get_page_marker do
body_path xpath(response, "//PageToken")
end
set_page_marker do
query "NextToken"
end
end
This is the same paginator except for JSON data.
pagination "aws_pagination_json" do
get_page_marker do
body_path jmes_path(response, "PageToken")
end
set_page_marker do
query "NextToken"
end
end
The following service has a X-Page-Token header that it both returns and expects:
pagination "my_service" do
get_page_marker do
header "X-Page-Token"
end
set_page_marker do
header "X-Page-Token"
end
end
The following service contains a nextLink parameter in the body that is a full URL to the next page of results. It is assumed that the datasource will specify we want JSON encoding.
pagination "azure_arm_pagination_json" do
get_page_marker do
body_path jmes_path(response, "nextLink")
end
set_page_marker do
uri true
end
end
The following service contains a Link header that includes the full URL to the next page of results.
pagination "github_api" do
get_page_marker do
header "Link"
end
set_page_marker do
uri true
end
end
The following service is an example of an AWS service where the NextPageToken is retrieved and set in the body.
pagination "aws_body_pagination" do
get_page_marker do
body_path jmes_path(response, "NextPageToken")
end
set_page_marker do
body_field "NextPageToken"
end
end
Some APIs use page numbers rather than a token, this is a case where the power of jq comes in handy as it can express mathematical concepts. Here is an example where the JSON body contains the current page number, the number of items per page, and the total number of items:
{
"paging": {
"pageIndex": 1,
"pageSize": 100,
"total": 381
},
...
}
In this case, you would define the following pagination with a jq expression which will return the next page number unless we have reached the last page:
pagination "sonarqube_pagination" do
get_page_marker do
body_path jq(response, "if .paging.pageIndex * .paging.pageSize < .paging.total then .paging.pageIndex + 1 else null end")
end
set_page_marker do
query "p"
end
end
There are also APIs that use page numbers without specifying the page number in the body. In these cases, you can take advantage of the $marker variable which is passed in when a jq expression is evaluated for a pagination. This $marker variable contains the string value of the previous pagination marker (for the first page, the previous page marker is an empty string: ""). Here is an example of the JSON that one of these APIs might have:
{
"items": [
...
]
}
Here is a pagination that will work with this API given that there are 500 items per page:
pagination "500_items_per_page_pagination" do
get_page_marker do
body_path jq(response, "if .items | length < 500 then null else if $marker != "" then $marker | tonumber + 1 else 2 end end")
end
end
Just like the previous pagination, this one also uses null to signify that we have reached the last page.
The syntax of a datasource that retrieved data from an HTTP API is:
datasource <name> do
request do
auth $<credentials reference OR auth definition>
pagination $<pagination definition>
host <host>
verb ["GET"|"POST"] # defaults to "GET"
scheme ["http"|"https"] # defaults to "https"
path <path>
ignore_status [<http status code>, <http status code>...]
query <query string name>, <query string value>
query <query string name>, <query string value>
...
header <header name>, <header value>
header <header name>, <header value>
...
body_field <body field name>, <body field value>
body_field <body field name>, <body field value>
...
body <body value>
end
result do
encoding ["json"|"xml"|"text"]
[collect jmes_path(response, <jmes_path>) do]
[collect jq(response, <jq>) do]
[collect xpath(response, <xpath>) do]
field <field name>, <term>
field <field name>, <term>
...
[end]
end
end
Where:
| • | <name> is the name of the datasource. |
| • | <auth definition> is the name of the credentials reference or auth definition that describes how to authenticate requests made to the API. |
| • | <pagination definition> is the name of the pagination definition that describes how to iterate through the response pages if any. |
| • | <host> is the API hostname. |
| • | <verb> is the HTTP request method and defaults to "GET". |
| • | <path> is the HTTP path |
| • | <ignore_status> are http status codes to ignore failure on. Normally any status code other than 200-299 will cause the policy to stop evaluation and return a failure. Supply a list of values or a list parameter such as [403, 404] to cause the policy to treat these calls as an empty dataset and continue running. |
| • | <query string name> and <query string value> describe the request query string elements if any. |
| • | <header name> and <header value> describe any additional headers you wish to send with the API request. |
| • | <body field name> and <body field value> describe the body JSON object fields if any. When a body_fieldis specified, body may not be specified as well. |
| • | <body value> describes the entire body. When body is specified, no body_fields may be specified as well. |
Note:Only JSON encoding is supported in the REQUEST body at this time when using body_field. However, other encodings can be sent using body.
The syntax of a JavaScript datasource that retrieved data from an HTTP API is:
script <name>, type: "javascript" do
result "results"
code <<-EOS
results = {
"auth": $<credentials reference OR auth definition>,
"pagination": $<pagination definition>,
"host": <host>,
"verb": ["GET"|"POST"] # defaults to "GET",
"scheme": ["http"|"https"] # defaults to "https",
"ignore_status": [<http status code>, <http status code>...],
"path": <path>,
"headers": {
"<header name>": "<header value>",
...
},
"query_params": {
'<query string name>': '<query string value>',
...
},
"body": <body value>
}
EOS
end
You can use the Javascript request to customize your request with different options during runtime.
Fields:
| • | <name> is the name of the datasource. |
| • | <auth definition> is the name of the credentials reference or auth definition that describes how to authenticate requests made to the API. |
| • | <pagination definition> is the name of the pagination definition that describes how to iterate through the response pages if any. |
| • | <host> is the API hostname. |
| • | <verb> is the HTTP request method and defaults to "GET". |
| • | <path> is the HTTP path |
| • | <ignore_status> are http status codes to ignore failure on. Normally any status code other than 200-299 will cause the policy to stop evaluation and return a failure. Supply a list of values or a list parameter such as [403, 404] to cause the policy to treat these calls as an empty dataset and continue running. |
| • | <query string name> and <query string value> describe the request query string elements if any. |
| • | <header name> and <header value> describe any additional headers you wish to send with the API request. |
| • | <body value> describes the entire body. When body is specified, no body_fields may be specified as well. |
Example:
First, create a request block around the run_script command.
datasource "ds_resources_by_date" do
request do
run_script $js_resources_by_date
end
result do
encoding "json"
...
end
end
end
Second, create a script that returns a request object.
script "js_resources_by_date", type: "javascript" do
result "results"
parameter "region"
code <<-EOS
var end_date = new Date().toISOString()
var start_date = new Date(new Date().setDate(new Date().getDate() - 30)).toISOString();
results = {
"auth": "auth_aws",
"host": 'ec2.'+region+'.amazonaws.com',
"verb": "GET",
"path": "/",
"headers": {
"User-Agent": "RS Policies",
"Content-Type": "text/xml"
}
"query_params": {
'Action': 'DescribeInstance',
'ResourceName': instanceArn,
'Version': '2014-10-31',
'StartDate': start_date,
'EndDate': end_date
}
}
EOS
end
The result property details how to parse the HTTP response to produce the resulting data. The properties are:
| • | encoding—specifies the response encoding, must be "json", "xml", or "text". With the "text" encoding the body of the response will be returned as a string or an array of strings if multiple requests are made (either in the case of iteration or pagination). |
| • | field—identifies a field in the resulting data. The provided term is evaluated to initialize the field value. Typically the term is either the jmes_path, jq, or xpath function. |
result may also make use of collect to iterate over the response elements. collect accepts a term and a block. The term must be jmes_path, jq, or xpath. The term must return an array which collect iterates over. Each element of the array can then be used to define the data fields. The current element is accessed using the col_item reserved word as shown in the example below.
datasource "volumes_us_east" do
request do
auth $cred_aws_us_east_1
pagination $aws_pagination_xml
host "ec2.amazonaws.com"
query "Action", "DescribeVolumes"
query "Filter.1.Name", "encrypted"
query "Filter.1.Value.1", "false"
header "User-Agent" "My-app"
end
result do
encoding "xml"
collect xpath(response, "/DescribeVolumesResponse/volumeSet/item") do
field "id", xpath(col_item, "volumeId")
field "region", "us-east-1"
field "size", xpath(col_item, "size")
field "type", xpath(col_item, "volumeType")
field "tag_set" do
collect xpath(col_item, "tagSet") do
field "key", xpath(col_item, "key")
field "value", xpath(col_item, "value")
end
end
end
end
end