Troubleshooting Forcepoint ONE's Active Directory Sync Agent

Troubleshooting Forcepoint ONE's Active Directory Sync Agent

Summary
Solutions to common troubleshooting scenarios with AD sync Agent
Problem

The Bitglass DirSync AD Agent is a program that syncs Active Directory users to the Forcepoint ONE user database. An AD user (typically an admin) would download the tool and install it on a local machine. The agent runs and automatically syncs every hour. 

When one or more users are not syncing from AD, below are some common causes and remedies. 

Solution

If these steps do not fully resolve the issue, please get in touch with Forcepoint ONE technical support and provide the Email / UPN of the user(s) who are not being synced successfully, and any troubleshooting steps already taken. 
 

Prerequisites:

  1. Forcepoint ONE / Bitglass AD Agent must be deployed and configured properly to import users from AD into Forcepoint ONE, following the steps in the article How to install Dirsync/ AD sync service on Windows Server
  2. Determine which AD value is being used to sync the relevant users: either User Principle Name (UPN) or Email address
    • In the Bitglass portal, under IAM > Users and Groups, click on the relevant domain name under "Active Directory User Source
    • Confirm whether "Login Username" is configured to sync users by their UPN or Email.
  3. How to initiate a sync from Bitglass
    • In the Forcepoint ONE portal navigate to IAM Users and Groups, and click the link for the relevant domain under Active Directory User Source
  • Start Sync - Will initiate on-demand sync for any changes made in Active Directory since the last agent synchronization. This sync occurs automatically every 3 hours. 

  • Start Full Sync - Initiates a full synchronization rather than synchronizing only changes in AD.

In the below steps, unless otherwise specified, we will be using the "Start Sync" option.

 

Common Issues with AD Sync Agent

User Flagged for Deactivation
AD Sync Agent Misconfiguration Issues
No Users Syned From a Particular Group
User’s AD Profile Is Missing a Required Attribute
User’s Domain Is Not Selected for Syncing
AD Agent Unable To Query Active Directory
Newly Added AD Users Not Synced to Forcepoint ONE
Authentication errors due to AD Agents' version mismatch
Advanced Troubleshooting in AD(LDAP)
Frequently Asked Questions
 

  1. User Flagged for Deactivation:

The AD agent maintains users and notifies the admin when a user needs to be deactivated. The AD Sync Agent has discovered a user discrepancy and has flagged the user to be deactivated. As this configuration is set to require manual deactivation, a log message is displayed in the GUI:

Example Log: Agent sync event recommends User BG User4 (bguser4@xxxx.xx) to be deactivated. Please confirm user deactivation under IAM > Users and Groups > All Users.


Analyze > Logs > Admin:


IAM > Users and Groups > All Users > Search for the user

You may choose to either click on Confirm Deactivation or investigate why the deactivation was flagged.  The AD Sync Agent queries all LDAP objects/groups you configured every 3 hours. It then does a diff against the previous query. Anything that is different is sent to the frontend. The frontend then compares what it currently has versus the new information. If the ADSA diff file noted a user is no longer in a group, it will cause the frontend to flag the user as needing deactivation, hence, the notification is triggered.  You can also configure the frontend to remove the user automatically (configured in AD Sync Agent)

 

  1. AD Sync Agent Misconfiguration Issues :

Forcepoint ONE portal shows the AD Sync Agent is configured, but it has never connected to it, or the health check is down.
  • Login to the admin portal as a System Administrator. Go to IAM > Users and Groups Active Directory User Source (click on the domain). Check for the last sync time if one has occurred previously:

If anything shows up here, then it has connected at least once.
  • If not, then it has never connected. Login to the server which has the agent installed and open the console. Verify the service is running:

  • If it is not running, you will see a yellow notification:

  • If the password is incorrect for the Agent to Forcepoint ONE portal connection, you will also see a yellow notification:

  • Lastly, if the AD connection was successful based on the information entered in the console, but the machine is not on the domain (this is a requirement), then you will get all green status marks. However, the Event Viewer will show a 5XX series error indicating AD Sync Agent failed to get the correct data back from the AD.

All of these types of errors are user misconfigurations and can be corrected by updating the configuration settings either on the Agent or in the portal.


 

  1. No Users Syned From a Particular Group

  • Login to the admin portal as a System Administrator. Navigate to IAM > Users and Groups Active Directory User Source (click on the domain). Check that the users and groups in question are check marked to be synced:

  • Login to the server the agent is installed and open the event viewer. Check for 202 errors denoting issues with the group sync:

The group must be marked to sync and should show a count of users that have successfully synced. If the group is check marked and shows 0 users, either there are no users in that group, or the AD Sync Tool had an issue getting data from the AD server.
 

  1. User’s AD Profile Is Missing a Required Attribute

  • Find the users that are not being synced in your Windows Active Directory and check their UPN or Email fields, depending on which option you've selected from the above prerequisites. 
  • If the relevant value is empty, add the required information in Active Directory and initiate a sync. 
  1. User’s Domain Is Not Selected for Syncing

A user will only be synced if the domain is registered in Forcepoint ONE.
  • In the Forcepoint ONE portal, under IAM Users and Groups, under the "Username Domain" section, confirm whether the user's Email / UPN domain is listed.
  • If the correct domain is not already present, click the green plus (+) icon and add the user's domain name and select the appropriate authentication method for those users. Then initiate a sync from the Forcepoint ONE portal. 
  • The absence of a domain in the portal may cause synchronization errors for users, such as: "502 Error - Error sending payload to Bitglass"
  1. AD Agent Unable To Query Active Directory

  • Log in to the windows machine running the Bitglass AD Sync agent and run the Bitglass DirSync console
  • The console will indicate whether it is able to successfully connect to your AD server
 
 
To troubleshoot the AD Connection failure, there are a few tests you can run to narrow down where the issue is occurring:

 

  • Confirm DNS resolution to the AD Server with nslookup
Run Command Prompt from the same windows machine. 
For example, based on the above screenshot, run the command:
nslookup ad.example.com
If this command does not return an IP address, validate your DNS configuration. 
  • Confirm connectivity to the configured server and port. 
For example, based on the above screenshot, run the command:
 
telnet ad.example.com 389
Note: You may need to enable the telnet client

If the nslookup command returns multiple IP addresses, run the telnet command against each of those IP addresses, one at a time, like:
telnet 10.1.2.3 389

If the telnet connection fails for any of the specified IPs, troubleshoot your network and relevant firewall rules, or update the server name and/or port correctly, then start the Bitglass DirSync Service. 

 

  • Confirm that the domain added in the Domain Name field is a Fully Qualified Domain Name (FQDN) or an IP address.
  • Add domain name in lowercase
  • Try adding AD Admin user with the format Domain Name \ User Name

 

  1. Newly Added AD Users Not Synced to Forcepoint ONE

  • On the server running the Bitglass sync agent, open Windows Explorer and navigate to C:\ProgramData\bitglass\dirsync
  • Sort by Date modified and open the most recent snapshot-[timestam].xml 
  • Search the snapshot XML for the relevant email address or UPN of the user not being synced. 
  • If the user is present in the snapshot, initiate a Full Sync from the Bitglass portal 

 

  1. Authentication errors due to AD Agents' version mismatch

Users encounter authentication failure when using AD Agent for authentication even though the password entered is accurate.
AD Agent authentication fails even though AD Agents are healthy and connected in the Forcepoint ONE portal.

Two or more AD Agents instances are recommended when using an AD Agent for authentication. When the major version of these AD Agents are mismatched, that is, one agent is 1.xx.x.x while the other is 2.xx.x.x, authentication fails even though the password from the user is accurate.

AD Agents retrieve Authentication task messages from Forcepoint ONE servers and responds with success or failure results for them. The format of these authentication messages differs depending on the AD Agent version installed. AD Agents with lower major versions (e.g. 1.xx.x.xxx) cannot decipher messages meant for higher major versions (e.g. 2.xx.x.xxx). This leads to authentication failures for end users even with accurate passwords when AD Agent major versions are mismatched in an environment.

 

How to Identify AD Agent version mismatch:

 

On the Forcepoint ONE portal, the status of the AD Agents installed can be viewed by navigating to IAM Users and Groups > Active Directory User Source

The screenshot below shows AD Agent installations that have the problematic mismatched major versions:

 

Solution:

Ensure all AD Agents have the same major version and that two or more AD Agents are running and healthy in your AD infrastructure.
 

  • Click the Download agent button to download the most current version.
  • One at a time, run the installer on each server running the agent to upgrade. Please wait at least 30 minutes between each upgrade. 
  • Checking the agents again, ensure all agents are currently running the same version. 

The picture below shows two healthy AD Agents running the same major version and providing successful user authentication.

 

Advanced Troubleshooting in AD(LDAP)

These are the two tools that can help you find the LDAP queries the Bitglass sync agent is sending to Forcepoint ONE servers.

  1. AD Insight - Below is the download link for this tool 
  2. AD Explorer - Below is the download link for this tool

Download these tools on the machine which is running the AD agent. 



Frequently Asked Questions :

  • How does the AD Agent/Frontend figure out new information (like changed users, new users, etc.)

The AD agent makes a .Net (LDAP) request to the domain controller for the groups that the Admin/System Admin specified during setup.  It records these in a snapshot file (snapshot-<date>). It then runs a Diff on the previous snapshot and any changes are sent to the frontend over REST HTTP calls. The frontend then compares the list against what it has and makes the necessary changes and notifications.

  • Where are the Snapshot and Diff files stored?

The AD Agent keeps these files in C:/programdata/bitglass/dirsync on the machine it is installed on. Keep in mind, this is a hidden folder. You can open up a system window and type in the exact location and it will open.

  • How do I launch the GUI that shows the service status?

Start > Programs > Bitglass Agent Console. The console is only a viewer, the service is running even after you close the console window.  The console itself is used to start/stop the service manually and update LDAP or Bitglass login information.

  • How do I troubleshoot the AD Sync Agent on the machine it is installed on?

You can look at event logging. Go to Start->run->eventvwr.msc and hit enter. This will launch the event viewer. Go into Windows Logs->Applications and you will see the Bitglass Agent logging. This article explains the different error codes and what they denote Directory Sync AD Agent Event Codes

  • How do I enable debugging to get more event logging?

Open the console and stop the service. Open up C:/programdata/bitglass/dirsync/config.xml and set 'false' to 'true' in the debug section of the config file. Go back to the console and start the service again. More logging will show up in the event viewer. Be sure to shut this off as soon as possible as it is very verbose.

 

  • How often is Sync done automatically and how often does the agent report to the frontend?

Keepalives are sent every 60 seconds, periodic syncs are done every 3 hours. As part of the keepalive, the agent is actually querying if the frontend has anything for it to do. If the admin has triggered a full sync or partial sync by clicking the button in the portal, then the frontend will tell the next agent to start a sync through the keepalive response (over JSON).

The agent will check with the AD server every 3 hours for any changes and will update the frontend at the same time. So it may take up to 3 hours for AD changes to reflect on the Forcepoint ONE portal unless manual sync is done from the portal.

 

Further Information:

How to install Dirsync/ AD sync service on Windows Server
Deploying an AD Agent
Directory Sync AD Agent Event Codes
AD Agent - Using an upstream proxy to connect to Forcepoint ONE portal