The ingestion system consists of three main components:
Microsoft Graph API Client - Handles OAuth 2.0 authentication and fetches user data including manager information
Jira Cloud Assets Client - Interacts with Jira Service Management Assets API using workspace tokens
Ingestion Engine - Coordinates entire process including user matching, change detection, object lookups, and update logic
The system processes users in a streaming fashion:
Fetches users from Microsoft Graph API
Matches them against existing assets using multiple criteria
Detects changes while preserving existing data
Looks up object references (Company, Department, License)
Updates or creates users with proper error handling
Key Challenges Addressed
Atlassian's Jira Service Management (JSM) Assets API lacks comprehensive public documentation for certain operations. This implementation provides practical solutions for:
Multiple cache maps are implemented for performance and fallback support:
javascript
1// ID-to-name mapping - enables attribute lookups by name2this.attributeIdToNameMap=newMap();3this.attributeIdToTypeInfoMap=newMap();45// Object lookup caches - stores ALL potential matches for fallback6this.companyObjectCache=newMap();7this.departmentObjectCache=newMap();8this.licenseObjectCache=newMap();9this.locationObjectCache=newMap();1011// User lookup cache - avoid repeated Jira REST API calls12this.userObjectCache=newMap();
Understanding Jira Assets Attribute Types
Jira Assets uses three main attribute types that require different handling:
Type 0 (Text/String Attributes)
Simple string values like Name, Email, Title, Location (as text). Direct read/write with string values. No special handling required.
Type 1 (Object Reference Attributes)
References to other Assets objects like Company, Department, License. Most complex to handle!
Problem 1: AQL returns minimal information When querying Assets with AQL, response only includes objectTypeAttributeId and NOT the full attribute object with name field.
Example AQL Response Structure:
json
1{2"objectEntries":[3{4"id":"12345",5"objectTypeAttributeId":"1486",6"objectAttributeValues":[7{8"value":"UD-14341",9"displayValue":"Example Company Inc."10}11]12}13]14}
Note: The objectTypeAttributeId (1486) tells us this is a Company attribute, but we don't know the attribute NAME without additional lookup.
Problem 2: Different values needed for comparison vs. update
For COMPARISON: We need displayValue (object name) to match against Microsoft Graph data
For UPDATE: We need value (object ID/key) to write to asset
Example Asset State:
json
1{2"attributes":[3{4"objectTypeAttributeId":"1486",5"objectAttributeValues":[6{7"value":"UD-14341",// This is the ID we write8"displayValue":"Example Company Inc."// This is the name we compare9}10]11}12]13}
Type 2 (User Attributes)
References to Jira Platform users by accountId. The challenge:
Problem: Email to AccountId Mapping Microsoft Graph provides email addresses (e.g., user@example.com), but Jira Assets requires accountId (e.g., 70121:abc-123-def) for User type attributes.
Solution: Separate Jira REST API Lookup We need to call Jira REST API to look up user by email and retrieve their accountId.
Solution 1: ID-to-Name Mapping Cache
Since AQL responses only return objectTypeAttributeId, we need to build a cache that maps attribute IDs to their names. This enables us to look up attributes by name instead of ID.
Implementation Code
javascript
1// First, fetch all attributes and build ID-to-name mapping2const attributes =awaitthis.cloudApiClient.getObjectTypeAttributes(3this.options.objectTypeId4);56this.attributeIdToNameMap=newMap();7this.attributeIdToTypeInfoMap=newMap();89for(const attr of attributes){10this.attributeIdToNameMap.set(attr.id, attr.name);11this.attributeIdToTypeInfoMap.set(attr.id,{12name: attr.name,13type: attr.type,// Type codes: 0=Text, 1=Object, 2=User14});15}
Using the Cache
javascript
1getAttributeValue(asset, attributeName){2// Find attribute by ID using ID-to-name mapping3const attr = asset.attributes.find((a)=>{4const attrName =this.attributeIdToNameMap[a.objectTypeAttributeId];5return attrName && attrName.toLowerCase()=== attributeName.toLowerCase();6});78if(!attr ||!attr.objectAttributeValues||9 attr.objectAttributeValues.length===0){10return"";11}1213return attr.objectAttributeValues[0].value;14}
Type 1 Attributes - Return DisplayValue for Comparison
For Object Reference attributes, we need to return displayValue instead of value for comparison:
For Type 1 attributes like Company, Department, License, we need to look up the object by name and use its ID for updates.
Implementation Code
javascript
1asynclookupObjectId(attributeName, value){2if(!value){3return"";4}56// Cache lookup to avoid repeated AQL queries7const cacheKey =`${attributeName}:${value}`;89// Check if we have cached potential matches10if(this.companyObjectCache.has(cacheKey)){11const cachedMatches =this.companyObjectCache.get(cacheKey);1213// Return first match if cached as array14if(Array.isArray(cachedMatches)&& cachedMatches.length>0){15return cachedMatches[0];16}1718// Return single value if cached as string19if(typeof cachedMatches ==="string"){20return cachedMatches;21}22}2324console.log(`Looking up ${attributeName} object for: ${value}`);2526try{27const normalizedAttrName = attributeName.toLowerCase();2829// Auto-discover Company object type if needed30let objectTypeId =null;31if(normalizedAttrName ==="company"&&!process.env.COMPANY_OBJECT_TYPE_ID){32console.log(`Auto-discovering Company object type...`);33const schemas =awaitthis.cloudApiClient.getSchemas();34for(const schema of schemas){35const objectTypes =awaitthis.cloudApiClient.getObjectTypes(schema.id);36const companyType = objectTypes.find(37(ot)=> ot.name.toLowerCase()==="company"38);39if(companyType){40 objectTypeId = companyType.id;41console.log(`Found Company object type: ${companyType.name} (ID: ${objectTypeId})`);42break;43}44}45}4647// Build AQL query to find object by name OR key48const escapedValue =this.escapeAQLValue(value);49let aqlQuery =`Name = "${escapedValue}" OR Key = "${escapedValue}"`;5051// Add objectType filter for Company/Location/License (not Department)52if(objectTypeId && normalizedAttrName !=="department"){53 aqlQuery +=` AND objectType = ${objectTypeId}`;54}5556console.log(`AQL Query: ${aqlQuery}`);5758// Search for multiple matches (limit 10 instead of 1) to allow validation59const response =awaitthis.cloudApiClient.executeAQL(60 aqlQuery,610,6210,63true,64);6566if(response &&(response.objectEntries|| response.values)){67const objectEntries = response.objectEntries|| response.values;68console.log(`Object entries found: ${objectEntries.length}`);6970if(objectEntries.length>0){71// CACHE ALL potential matches BEFORE validation for fallback!72const allPotentialMatches = objectEntries.map(73(obj)=> obj.key|| obj.objectKey,74);7576this.companyObjectCache.set(cacheKey, allPotentialMatches);7778// Validate each match and return first valid one79for(const object of objectEntries){80if(this.validateObjectMatch(attributeName, value, object)){81const objectKey = object.key|| object.objectKey;82console.log(`Validated and found ${attributeName}: ${object.label|| object.name} (Key: ${objectKey})`);83return objectKey;84}85}86}87}8889return"";// No object found90}catch(error){91console.warn(`Error looking up ${attributeName}: ${error.message}`);92return"";93}94}
Why This Matters - Business Rule Restrictions
Multiple Company objects might have the same name but different keys due to different schemas or object types:
Object entries found: 2 Match 1: Example Company (Key: UD-14341) Match 2: Example Company (Key: ADI-2952)
Problem: Some Company objects cannot be used with certain User object types due to business rule restrictions.
Solution: Cache ALL potential matches as an array. If the first match fails due to restrictions, we can automatically try the next match!
Solution 3: User Type Attribute Lookup
Jira User attributes reference Jira Platform users by accountId. This requires a separate API call to Jira REST API.
Implementation Code
javascript
1asynclookupUserAccountId(email){2// Check cache first to avoid repeated REST API calls3if(this.userObjectCache.has(email)){4returnthis.userObjectCache.get(email);5}67console.log(`Looking up Jira User for email: ${email}`);89try{10// Search for Jira Platform user by email11const jiraUser =awaitthis.cloudApiClient.searchJiraUser(email);1213if(jiraUser && jiraUser.accountId){14// Cache the result15this.userObjectCache.set(email, jiraUser.accountId);1617console.log(`Found Jira User: ${jiraUser.accountId} (${jiraUser.displayName})`);18return jiraUser.accountId;19}else{20console.warn(`Jira User not found for email: ${email}`);21return"";22}23}catch(error){24console.warn(`Error looking up Jira User: ${error.message}`);25return"";26}27}
Usage in Update
javascript
1// In updateUser(), handle Type 2 attributes2if(attrInfo.attributeType===2){3 convertedValue =awaitthis.lookupUserAccountId(change.newValue);45// Skip User type attributes if lookup failed (user not found in Assets)6if(!convertedValue || convertedValue ===""){7console.log(`Skipping ${attrInfo.attributeName} - user not found in Assets`);8continue;9}10}1112// Then use the accountId for update13updatedAttributes.push({14objectTypeAttributeId: attrInfo.attributeId,15objectAttributeValues:[{value: convertedValue }],16});
Why This Matters
Different APIs with Different Rate Limits:
Microsoft Graph API uses OAuth with its own rate limits
Jira REST API has different rate limits than AQL-based Assets API
Caching Jira User lookups is critical to avoid hitting REST API rate limits
Solution 4: Data Preservation Strategy
Microsoft Graph is treated as the authority, but we should NEVER clear existing valid data from Assets. If Microsoft Graph doesn't have a value but the asset does, we MUST preserve the existing value.
Microsoft Graph has: companyName: "Example Company Inc."
Result: Change detected, will update
Scenario 3: Don't Clear Valid Company
Asset has: Company: "Example Company Inc."
Microsoft Graph has: companyName: ""
Result: No change detected, data preserved (NOT CLEARED!)
Solution 5: Error Handling & Fallback Mechanism
When updating assets, we might encounter errors like business rule restrictions or validation errors. The fallback mechanism removes failing attributes and retries with remaining ones.
Complete Implementation
javascript
1asyncupdateUser(asset, graphUser, changes){2console.log(`Updating asset ${asset.objectKey|| asset.id}`);34// Fetch existing object's current attributes to preserve them5let existingAttributes ={};6try{7const existingObj =awaitthis.cloudApiClient.getObject(asset.id);8if(existingObj && existingObj.attributes){9for(const attr of existingObj.attributes){10 existingAttributes[attr.objectTypeAttributeId]= attr;11}12}13}catch(error){14console.warn(`Warning: Could not fetch existing attributes: ${error.message}`);15}1617// Build attributes to update - merge new values with existing ones18const updatedAttributes =[];1920// First, copy all existing attributes to preserve them21for(const[attrId, attrValue]ofObject.entries(existingAttributes)){22 updatedAttributes.push({23objectTypeAttributeId:parseInt(attrId),24objectAttributeValues: attrValue.objectAttributeValues,25});26}2728const failedAttributes =[];2930// Then, update with new values for changed attributes31for(const change of changes){32const attrInfo =this.attributeMapping[change.graphAttribute];33if(attrInfo){34let convertedValue =this.convertValue(35 change.newValue,36 attrInfo.attributeType,37);3839// Handle Object type attributes (like Department, License, Location, Company)40if(attrInfo.attributeType===1){41 convertedValue =awaitthis.lookupObjectId(42 attrInfo.attributeName,43 change.newValue,44);45// Skip Object type attributes if lookup failed46if(!convertedValue || convertedValue ===""){47console.log(`Skipping ${attrInfo.attributeName} - referenced object not found in Assets`);48continue;49}50}5152// Handle User type attributes (like "Jira user")53if(attrInfo.attributeType===2){54 convertedValue =awaitthis.lookupUserAccountId(change.newValue);55// Skip User type attributes if lookup failed56if(!convertedValue || convertedValue ===""){57console.log(`Skipping ${attrInfo.attributeName} - user not found in Assets`);58continue;59}60}6162// Update or add the attribute63const attrIndex = updatedAttributes.findIndex(64(a)=> a.objectTypeAttributeId=== attrInfo.attributeId,65);6667if(attrIndex >=0){68// Update existing attribute69 updatedAttributes[attrIndex].objectAttributeValues=[70{value: convertedValue },71];72}else{73// Add new attribute74 updatedAttributes.push({75objectTypeAttributeId: attrInfo.attributeId,76objectAttributeValues:[{value: convertedValue }],77});78}79}80}8182const objectData ={83objectTypeId:this.options.objectTypeId,84attributes: updatedAttributes,85};8687if(this.options.debug){88console.log("DEBUG: Sending to updateObject:");89console.log(` objectId: ${asset.id}`);90console.log(` objectTypeId: ${objectData.objectTypeId}`);91console.log(" attributes:");92 updatedAttributes.forEach((attr)=>{93console.log(`${attr.objectTypeAttributeId}: ${JSON.stringify(attr.objectAttributeValues[0].value)}`);94});95}9697// Keep trying until success or no more attributes to update98while(updatedAttributes.length>0){99try{100const result =awaitthis.cloudApiClient.updateObject(101 asset.id,102 objectData,103);104105if(result){106console.log(`Updated successfully (${result.objectKey|| result.id})`);107108// Log failed attributes if any109if(failedAttributes.length>0){110console.log(`${failedAttributes.length} attribute(s) skipped due to errors:`);111 failedAttributes.forEach((failed)=>{112console.log(` - ${failed.attributeName}: ${failed.reason}`);113});114115// Add to global error list for reporting116this.errors.push({117type:"attribute_update_failed",118user: asset.label|| asset.objectKey,119userId: asset.id,120failedAttributes: failedAttributes,121});122}123124return result;125}126}catch(error){127// Try to identify which attribute caused the error128let failedAttrId =null;129let failedAttrName ="unknown";130let failureReason = error.message||"Unknown error";131132if(error.response){133let errorObj = error.response;134135// Try to parse JSON response136if(typeof error.response==="string"){137try{138 errorObj =JSON.parse(error.response);139}catch(e){140 errorObj = error.response;141}142}143144// Look for errors object in parsed JSON145const errorsObj = errorObj.errors|| errorObj;146147if(typeof errorsObj ==="object"&& errorsObj !==null){148// Iterate through error keys to find attribute restriction errors149for(const[key, value]ofObject.entries(errorsObj)){150const attrMatch = key.match(/^rlabs-insight-attribute-(\d+)$/);151if(attrMatch && attrMatch[1]){152 failedAttrId =parseInt(attrMatch[1]);153 failureReason = value ||"Invalid due to restrictions";154break;155}156}157}158}159160// Find the corresponding change161const failedChange = changes.find((c)=>{162const attrInfo =this.attributeMapping[c.graphAttribute];163return attrInfo && attrInfo.attributeId=== failedAttrId;164});165166if(failedChange){167console.log(`Failed to update ${failedChange.attributeName}: ${failureReason}`);168console.log(`Removing this attribute and retrying...`);169170// Track the failed attribute171 failedAttributes.push({172attributeId: failedAttrId,173attributeName: failedChange.attributeName,174reason: failureReason,175oldValue: failedChange.oldValue,176newValue: failedChange.newValue,177});178179// Remove this attribute from update180const attrIndex = updatedAttributes.findIndex(181(a)=> a.objectTypeAttributeId=== failedAttrId,182);183if(attrIndex >=0){184 updatedAttributes.splice(attrIndex,1);185}186187// Remove this change188const changeIndex = changes.indexOf(failedChange);189if(changeIndex >=0){190 changes.splice(changeIndex,1);191}192193// Rebuild objectData with remaining attributes194 objectData.attributes= updatedAttributes;195196// Try again with remaining attributes197continue;198}else{199// If we can't identify the failing attribute, log full error200console.error(`Could not identify which attribute caused the error. Full error: ${error.message}`);201console.error(`Error response: ${error.response||"No response"}`);202throw error;203}204}205}206207thrownewError(208`Failed to update all attributes for ${asset.objectKey|| asset.id}. See logs for details.`209);210}
Error Response Parsing
The code handles both string and object error responses:
javascript
1// If error.response is a JSON string2if(typeof error.response==="string"){3try{4 errorObj =JSON.parse(error.response);5}catch(e){6 errorObj = error.response;7}8}910// Parse errors object11const errorsObj = errorObj.errors|| errorObj;1213// Iterate through error keys to find attribute restriction errors14for(const[key, value]ofObject.entries(errorsObj)){15const attrMatch = key.match(/^rlabs-insight-attribute-(\d+)$/);16if(attrMatch && attrMatch[1]){17 failedAttrId =parseInt(attrMatch[1]);18 failureReason = value ||"Invalid due to restrictions";19}20}21
How the Fallback Works
First Attempt - Try to update all attributes
Error Encountered - API returns validation error for attribute 1486 (Company)
Parse Error - Extract attribute ID from "rlabs-insight-attribute-1486"
Identify Failed Change - Find the change that maps to attribute 1486
1const aqlQuery =`objectType = ${this.options.objectTypeId}`;23const response =awaitthis.cloudApiClient.executeAQL(4 aqlQuery,50,6500,// Batch fetch all existing users7true8);910// Build ID-to-name mapping cache from first sample entry11if(response.objectEntries&& response.objectEntries.length>0){12const sampleEntry = response.objectEntries[0];1314// Build mapping for attribute lookups15for(const attr of sampleEntry.attributes){16this.attributeIdToNameMap.set(attr.objectTypeAttributeId, attr.name);17this.attributeIdToTypeInfoMap.set(attr.objectTypeAttributeId,{18name: attr.name,19type: attr.type,20});21}2223// Build user lookup map24for(const entry of response.objectEntries){25if(entry.attributes){26const nameAttr = entry.attributes.find(27a=> a.objectTypeAttributeId===147828);29if(nameAttr && nameAttr.objectAttributeValues[0]){30const name = nameAttr.objectAttributeValues[0].value;31if(name){32this.existingUsersMap.set(this.normalizeKey(name), entry);33}34}35}36}37}38
Step 4: Fetch Users from Microsoft Graph
javascript
1const users =awaitthis.graphApiClient.fetchUsers({2filter:"userType eq 'Member'",3select:Object.keys(ATTRIBUTE_MAPPING).join(","),4maxResults:this.options.limit||0,5expandManager:true,// Expand to get manager information6});7
Step 5: Process Each User
javascript
1for(let i =0; i < users.length; i++){2const user = users[i];3const displayName = user.displayName|| user.id||"Unknown";4const mail = user.mail||"";5const userPrincipalName = user.userPrincipalName||"";6const managerInfo = user.manager7?`${user.manager.displayName} (${user.manager.mail||""})`8:"None";910console.log(`[${i +1}/${users.length}] Processing: ${displayName}`);11console.log(` Email: ${mail ||"N/A"}`);12console.log(` UPN: ${userPrincipalName}`);13console.log(` Manager: ${managerInfo}`);1415// Try to find matching existing user16const existingAsset =awaitthis.findMatchingUser(user);1718if(existingAsset){19this.stats.existingUsersMatched++;2021// Compare attributes to detect changes22const changes =this.detectChanges(user, existingAsset);2324if(changes.length===0){25console.log(`MATCHED: No changes detected`);26this.stats.usersSkipped++;27}else{28console.log(`MATCHED: ${changes.length} change(s) detected`);29 changes.forEach((change)=>{30let changeMessage =` - ${change.attribute}: "${change.oldValue}" → "${change.newValue}"`;3132// Add note for object reference attributes33if(change.attributeType===1){34 changeMessage +=` (will look up object by name and use its ID)`;35}3637console.log(changeMessage);38});3940if(!this.options.dryRun){41awaitthis.updateUser(existingAsset, user, changes);42this.stats.existingUsersUpdated++;43}else{44console.log(` [DRY RUN] Would update asset ${existingAsset.objectKey|| existingAsset.id}`);45this.stats.usersSkipped++;46}47}48}else{49console.log(`NEW: Creating as new asset`);5051if(!this.options.dryRun){52awaitthis.createUser(user);53this.stats.newUsersCreated++;54}else{55console.log(` [DRY RUN] Would create new asset`);56this.stats.usersSkipped++;57}58}59}
Step 6: Save Report
javascript
1saveReport(){2const logsDir = path.join(__dirname,"..","logs");3if(!fs.existsSync(logsDir)){4 fs.mkdirSync(logsDir,{recursive:true});5}67const timestamp =newDate().toISOString().replace(/[:.]/g,"-");8const reportPath = path.join(logsDir,`ingestion-report-${timestamp}.json`);910const report ={11timestamp:newDate().toISOString(),12options:this.options,13duration:this.stats.endTime-this.stats.startTime,14durationString:this.formatDuration(this.stats.endTime-this.stats.startTime),15stats:this.stats,16graphApiStats:this.graphApiClient.getStats(),17cloudApiStats:this.cloudApiClient.getStats(),18errors:this.errors,// Include all failed attributes for manual fix19};2021 fs.writeFileSync(reportPath,JSON.stringify(report,null,2));22console.log(`Report saved to: ${reportPath}`);23}
Best Practices
1. Always Use Caching
Cache all potential object matches, not just the first one. This enables fallback when business rules restrict certain objects.
2. Data Preservation is Critical
Never clear existing valid data from Assets. Only update when Microsoft Graph has a value and asset is empty.
3. Handle Rate Limits
Implement retry logic with exponential backoff for API calls. Jira REST API has different rate limits than AQL-based Assets API.
4. Comprehensive Error Logging
Track all failures in detailed error array with attribute IDs, names, reasons, and values. This enables manual review and fixes.
5. Test with Small Limits
Use --limit N flag to test with small batches before running full ingestion.
6. Use Dry-Run First
Always run --dry-run before actual ingestion to verify changes.
7. Monitor API Statistics
Track request counts and error rates to identify issues early.
Summary
This tutorial provides a production-tested solution for ingesting Microsoft Graph users into Jira Cloud Assets. The key learnings are:
Implement ID-to-Name Mapping - Enable attribute lookups by name instead of ID
Return Correct Values for Operations - displayValue for comparison, object ID for updates
Preserve Existing Data - Never clear valid data when Microsoft Graph is empty
Implement Fallback Mechanism - Handle business rule restrictions gracefully by removing failing fields and retrying
Log All Errors for Manual Fix - Track what couldn't be automated
By following this guide, you can build a robust ingestion system that handles complexities of Jira Assets API without relying on incomplete public documentation.
