Ansible Automation Platform Jobs Stuck in Pending: Root Cause and Fix


Today I ran into one of those issues that looks complex, but turns out to be beautifully simple.

Every job I launched in Ansible Automation Platform (Tower / Controller 4.5) just sat there.
No output. No errors. No movement. Just PENDING.

And sometimes, silence is the loudest signal.

What I Saw

Everything looked healthy on the surface:

  • Job templates launched successfully
  • Jobs stayed in PENDING forever
  • No logs, no failures, no hints
  • Execution Environments looked perfectly fine

At first, I suspected Ansible itself,  playbooks, environments, something deep.

But this didn’t feel like a playbook problem.
This felt like something wasn’t even starting.

The Turning Point

When jobs don’t start at all, it’s usually not Ansible… it’s scheduling.

So I went one level lower services.

That’s when I checked Receptor, the quiet engine behind job execution in AAP 4.x.

And there it was:

systemctl status receptor
● receptor.service - Receptor
   Loaded: loaded (/usr/lib/systemd/system/receptor.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/receptor.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Tue 2026-03-31 12:49:06 CDT; 4s ago
  Process: 946196 ExecStart=/usr/bin/receptor -c /etc/receptor/receptor.conf (code=exited, status=1/FAILURE)
 Main PID: 946196 (code=exited, status=1/FAILURE)

Mar 31 12:49:06 xxxx systemd[1]: receptor.service: Service RestartSec=100ms expired, scheduling restart.
Mar 31 12:49:06 xxxx systemd[1]: receptor.service: Scheduled restart job, restart counter is at 5.
Mar 31 12:49:06 xxxx systemd[1]: Stopped Receptor.
Mar 31 12:49:06 xxxx systemd[1]: receptor.service: Start request repeated too quickly.
Mar 31 12:49:06 xxxx systemd[1]: receptor.service: Failed with result 'exit-code'.
Mar 31 12:49:06 xxxx systemd[1]: Failed to start Receptor.

Now we were getting somewhere.

The Real Error

Running Receptor manually revealed the truth:

error opening Unix socket: could not acquire lock on socket file: no such file or directory

That one line explained everything.

What Actually Broke

Receptor relies on a Unix socket located here:

/var/run/receptor

Here’s the subtle part:

  • /var/run (linked to /run) is temporary
  • It gets cleared on reboot or system cleanup
  • The directory /var/run/receptor was simply… gone

And Receptor?
It doesn’t recreate it.

So it failed silently.
And when Receptor is down:

  • No execution capacity
  • No scheduling
  • Jobs stay in PENDING forever

No errors in UI, because nothing even reached that layer.

The Fix

Sometimes the fix feels almost too simple.

I recreated the directory:

mkdir -p /var/run/receptor
chown receptor:receptor /var/run/receptor
chmod 755 /var/run/receptor

Then restarted services:

systemctl start receptor
automation-controller-service restart

And just like that—

PENDING → RUNNING

The system came back to life.

Making It Permanent

Because /var/run is temporary, this would happen again after reboot.

So I made it persistent using systemd:

Create:

/etc/tmpfiles.d/receptor.conf

Add:

d /var/run/receptor 0755 receptor receptor -

Then apply:

systemd-tmpfiles --create

Now the directory is recreated automatically every time.

What This Taught Me

When everything looks fine, but nothing moves:

  • Don’t start with playbooks
  • Don’t chase UI clues
  • Go deeper

Sometimes the failure isn’t in automation…
It’s in the foundation that enables it.

A missing directory.
A silent service.
A system waiting for something that no longer exists.

Final Thought

Not every problem announces itself.

Some just sit there quietly…
like a job stuck in PENDING,
waiting for you to look where no one else does.

Jay

Comments

Post a Comment

Popular posts from this blog

ASM Integrity check failed with PRCT-1225 and PRCT-1011 errors while creating database using DBCA on Exadata 3 node RAC

Image
  Error: The error message PRCT-1225: failed to verify Oracle Automatic Storage Management (Oracle ASM) user credentials using the command 'asmcmd credverify' indicates that the asmcmd credverify command failed to verify the Oracle ASM user credentials. This can happen for a number of reasons, including: The user credentials are incorrect. The user credentials file is corrupted. The user does not have the necessary permissions to run the asmcmd credverify command. There is a problem with the Oracle ASM instance. To troubleshoot this error, you can try the following: Make sure that the user credentials are correct. You can do this by trying to log in to the Oracle ASM instance using the same credentials. Check the user credentials file for errors. You can do this by using a text editor to open the file and inspect the contents. Make sure that the user has the necessary permissions to run the asmcmd credverify command. You can do this by checking the user's permissi...
Read more

Lock Tables in MariaDB

Whether or not you need to lock tables while backing up MariaDB depends on a few factors, including the type of backup you are performing and the tools you are using. If you are performing a full backup , you should lock the tables to ensure that the backup is consistent. This is because a full backup copies all of the data in the database, including any data that is currently being modified by transactions. If you do not lock the tables, it is possible that the backup could contain inconsistent data. If you are performing a differential backup or an incremental backup , you may not need to lock the tables. Differential and incremental backups only copy the data that has changed since the previous backup. This means that the data is already consistent, so there is no need to lock the tables. However, some backup tools may require you to lock the tables, even if you are performing a differential or incremental backup. This is because these tools may need to modify the database in some ...
Read more

Life is beautiful

Image
Life brings unexpected tragedies that are beyond our control. It's heartbreaking to witness people facing circumstances they never chose. We react with tears and inevitably wonder, "What if that happens to me?" As humans, we naturally focus on the positive and avoid dwelling on such thoughts—and that's healthy. But this perspective has shaped how I approach life: Am I prepared for these scenarios? Are my loved ones protected if something goes wrong? If I were to die, I couldn't be there to tell them to stay strong—they'd have to face that grief alone. But I can give them financial security through proper planning, wills, and trusts. These are tangible ways to care for them even when I'm gone. Beyond that, what else can I control? I don't waste energy worrying about an uncertain past or future. Instead, I choose to spread peace, truly listen to others, and share genuine smiles. I embrace the principle of living fully while letting others do the same. ...
Read more