SailPoint IdentityNow - Debugging techniques for web services connectors

IdentityNow integration series

SailPoint's web services connector is the swiss-army-knife in most custom implementations. SailPoint boasts a catalogue of dozens of out-of-the-box connectors but sometimes we have homebrew applications or our application is not included in the catalogue. Fear not, because it's easier than you think thanks to the web services connector. In a nutshell, a web services connector is an empty connector that allows you to map any operation a connector may have to a series of REST/SOAP endpoints. The creation of a web services connector is an iterative process because we need to properly map inputs with outputs, and this can be complicated when we don't have a clear vision on what's going on under the bonet.

This article consists of all the tips and tricks I personally use when creating/debugging a web services connector. I hope it helps you create your own more efficiently.

Debugging techniques for web services connectors

Turn on debugging output, for real!

Most times you get an error for any connector operation there's little or no trace of the endpoint call that was sent. This is generally the most interesting part to figure out what was wrong. We need to know the endpoint path, parameters, method and payload of the request, and also all details of the response. This information can be found in /home/sailpoint/ccg.log file in the corresponding VA, but you need to properly configure cluster, VA and connector for it to appear. Let's see how it's done:

1. Enable cluster debugging:

2. Configure /home/sailpoint/ccg/log4j2.properties on the VA with debug level for connector.sdk.webservices logger:

logger.sdk.name = connector.sdk.webservices
logger.sdk.level = debug
logger.sdk.additivity = false
logger.sdk.appenderRef.rolling.ref = STDOUT
        

3. Restart ccg service on the VA to make the change effective:

sudo systemctl restart ccg
        

4. Patch your web service source with this payload:

{
  "op": "replace",
  "path": "/connectorAttributes/isDevelopmentMode",
  "value": true
}
        

That should be enough for you to see full details on requests and responses generated by your connector. Look out for entries like these in ccg.log:

Dumping request for troubleshooting purposes: ...

Dumping response for troubleshooting purposes: ...
        

Here's an example of a response dump:

"DEBUG","IdentityIQ":"8.0 Build 2f61d76cf47-20220429-140148","message":"==> Dumping response for troubleshooting purposes: HttpResponseWrapper [status=200, response=[{\"synced\":\"2022-06-07T03:31:51.289Z\",\"displayName\":\"Administrator\",\"description\":\"Full administrative access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Administrator\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5c3b0a91\",\"value\":\"ORG_ADMIN\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.257Z\",\"displayName\":\"Auditor\",\"description\":\"Auditor access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Auditor\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5c7c0a95\",\"value\":\"AUDITOR\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.275Z\",\"displayName\":\"Certifications\",\"description\":\"Certifications administrative access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Certifications\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5c5c0a93\",\"value\":\"CERT_ADMIN\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.264Z\",\"displayName\":\"Dashboard\",\"description\":\"Dashboard access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Dashboard\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5c6b0a94\",\"value\":\"DASHBOARD\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.280Z\",\"displayName\":\"Helpdesk\",\"description\":\"Helpdesk access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Helpdesk\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5c4b0a92\",\"value\":\"HELPDESK\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.252Z\",\"displayName\":\"Report Administrator\",\"description\":\"Report Administrator access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Report Administrator\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5c8e0a96\",\"value\":\"REPORT_ADMIN\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.246Z\",\"displayName\":\"Role Administrator\",\"description\":\"Role Administrator access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Role Administrator\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5ca30a97\",\"value\":\"ROLE_ADMIN\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.237Z\",\"displayName\":\"Role SubAdministrator\",\"description\":\"Role SubAdministrator access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Role SubAdministrator\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5ccd0a98\",\"value\":\"ROLE_SUBADMIN\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.228Z\",\"displayName\":\"Source Administrator\",\"description\":\"Source Administrator access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Source Administrator\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5ce60a99\",\"value\":\"SOURCE_ADMIN\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"},{\"synced\":\"2022-06-07T03:31:51.198Z\",\"displayName\":\"Source SubAdministrator\",\"description\":\"Source SubAdministrator access to IdentityNow\",\"source\":{\"name\":\"IdentityNow\",\"id\":\"2c91808676f45b3f017701bf5c0b0a8e\"},\"segmentCount\":0,\"segments\":[],\"privileged\":false,\"name\":\"Source SubAdministrator\",\"modified\":\"2022-06-06T16:18:46Z\",\"requestable\":false,\"attribute\":\"assignedGroups\",\"id\":\"2c91808676f45b3f017701bf5cf60a9a\",\"value\":\"SOURCE_SUBADMIN\",\"_type\":\"entitlement\",\"type\":\"entitlement\",\"_version\":\"v7\"}], headers={Transfer-Encoding=chunked, Access-Control-Expose-Headers=Retry-After,Connection,SLPT-Request-ID,Content-Length,Date,X-Zuul-ServiceId,Content-Type, X-Robots-Tag=noindex, Server=nginx, Cache-Control=no-cache, no-store, must-revalidate, Connection=keep-alive, Vary=Access-Control-Request-Headers, SLPT-Request-ID=9cd2d2eef54f4ef793419be62e256671, Date=Tue, 07 Jun 2022 07:15:37 GMT, Content-Type=application\/json;charset=utf-8}]","pipeline":"1266","@timestamp":"2022-06-07T07:15:37.367Z","thread_name":"pool-5-thread-1","metrics":"1266","region":"eu-west-2","AppType":"Web Services","Application":"IdentityNow Roles [source]","request_id":"2e8bc78b9aa5475e85b8d7e9bdba1509","CB_Type":"connector-bundle-webservices","queue":"stg01-euwest2-enterprise522-cluster-452","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
        

Did you catch the error? No? You're too slow... Jokes aside, this is stringified JSON within a JSON body and make it pretty difficult to read it raw. We'll a couple of options to improve readability next. Before we do, note there's also interesting information about how the response is processed in the logs. You'll see it when you do it yourself.

jq-based wrapper for ccg.log

I created this simple wrapper to improve readability of ccg.log contents, not only web services connectors. I run this command and reproduce the steps I want to debug. The output is simpler and cleaner. It's enough for me to understand what's going on. Feel free to use it and improve it. It's not perfect but it does the trick for me.

Command:

tail -f /home/sailpoint/log/ccg.log|sed -n '/^{/p'|jq '.message,.Application'|sed 's/^"https://;s/"$//;s/\\//g'|jq -R '. as $raw | try fromjson catch $raw'|sed 's/\\"/"/g'
        

Sample output:

"IdentityNow Roles [source]"
"==> Dumping response for troubleshooting purposes: HttpResponseWrapper [status=200, response=[{"synced":"2022-06-07T03:31:51.289Z","displayName":"Administrator","description":"Full administrative access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Administrator","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5c3b0a91","value":"ORG_ADMIN","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.257Z","displayName":"Auditor","description":"Auditor access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Auditor","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5c7c0a95","value":"AUDITOR","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.275Z","displayName":"Certifications","description":"Certifications administrative access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Certifications","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5c5c0a93","value":"CERT_ADMIN","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.264Z","displayName":"Dashboard","description":"Dashboard access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Dashboard","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5c6b0a94","value":"DASHBOARD","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.280Z","displayName":"Helpdesk","description":"Helpdesk access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Helpdesk","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5c4b0a92","value":"HELPDESK","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.252Z","displayName":"Report Administrator","description":"Report Administrator access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Report Administrator","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5c8e0a96","value":"REPORT_ADMIN","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.246Z","displayName":"Role Administrator","description":"Role Administrator access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Role Administrator","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5ca30a97","value":"ROLE_ADMIN","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.237Z","displayName":"Role SubAdministrator","description":"Role SubAdministrator access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Role SubAdministrator","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5ccd0a98","value":"ROLE_SUBADMIN","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.228Z","displayName":"Source Administrator","description":"Source Administrator access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Source Administrator","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5ce60a99","value":"SOURCE_ADMIN","_type":"entitlement","type":"entitlement","_version":"v7"},{"synced":"2022-06-07T03:31:51.198Z","displayName":"Source SubAdministrator","description":"Source SubAdministrator access to IdentityNow","source":{"name":"IdentityNow","id":"2c91808676f45b3f017701bf5c0b0a8e"},"segmentCount":0,"segments":[],"privileged":false,"name":"Source SubAdministrator","modified":"2022-06-06T16:18:46Z","requestable":false,"attribute":"assignedGroups","id":"2c91808676f45b3f017701bf5cf60a9a","value":"SOURCE_SUBADMIN","_type":"entitlement","type":"entitlement","_version":"v7"}], headers={Transfer-Encoding=chunked, Access-Control-Expose-Headers=Retry-After,Connection,SLPT-Request-ID,Content-Length,Date,X-Zuul-ServiceId,Content-Type, X-Robots-Tag=noindex, Server=nginx, Cache-Control=no-cache, no-store, must-revalidate, Connection=keep-alive, Vary=Access-Control-Request-Headers, SLPT-Request-ID=8143a4fc140840b2baac80694bc14726, Date=Tue, 07 Jun 2022 07:55:17 GMT, Content-Type=application/json;charset=utf-8}]"
        

You can then copy/paste fragments of your output in your favorite editor for JSON coloring and formatting. It may still look cluttered in the sample but trust me, it's much easier to read.

Sending logs out

You may have some kind of log management system you're familiar with and you may want to leverage it. You can take multiple approaches to this and it also depends on the managemet system of choice. A solution could be adding a remote syslog appender to /home/sailpoint/ccg/log4j2.properties. This is a more permanent solution to that I'm going to describe next, but definitely worth considering on a corporate environment.

I personally use Loggly from Solarwinds. I use their free tier because I only send data when debugging, so if you don't have any log management system already, this is a free option for this purposes. Loggly has an HTTP endpoint where you can send logs to. You can easily send the contents of ccg.log by running this pipeline:

tail -f /home/sailpoint/log/ccg.log|while read i; do curl -H "content-type:application/json" -d "$i" https://logs-01.loggly.com/inputs/<your ID>/tag/http/;done > /dev/null 2>&1
        

As long as it's active, logs will be sent to Loggly where you can search and filter from the web interface. There must be a way to make Loggly parse the JSON payload but, honestly, I haven't dedicated much time to this:

This is a general improvement to the raw input and if you can parse the JSON payload, much better. Let me know if you figure it out.

Pipe in the middle

And you thought I wasn't going to talk about Pipedream? Hehe, you can also use Pipedream for this for convenience. I created this simple workflow that just forwards any request to a target, gets the response and sends that response to the original caller. Just like a man in the middle attack but for your own good. Let's see how to set it up:

1. Deploy your workflow and configure the target system you want to forward your requests to:

2. Now you need to change your source endpoints to point to Pipedream workflow:

One configuration trick you may need to use. When configuring a connector with different base urls for its operations, you can set Base URL to https:// and complete the url on each operation, plus the corresponding path, accordingly.

Your source should behave just like before but, now, you can clearly view what the request and the response were. You could even tweak the forwarded request if needed:

Request:

Response:

Well, that's all for today. I hope these few tricks help you configuring your connectors more easily.