• v7.0
    • Versions
    • master

 

  • Install Akeneo PIM
    • Install Akeneo PIM for development with Docker
    • Install Akeneo PIM manually
      • System Requirements
      • System installation on Debian 11 (Buster)
      • System installation on Ubuntu 22.04 (Jammy Jellyfish)
      • Installing Akeneo PIM Community Edition (CE)
      • Installing Akeneo PIM Enterprise Edition (EE) with the Archive
      • Setting up the job queue daemon
      • Setting up the job scheduling
      • Setting up the Events API
    • How to customize the Dataset
    • How to Add Translation Packs
  • Upgrade Akeneo PIM projects
    • How to apply a patch?
      • How to apply a patch - Community Edition
      • How to apply a patch - Enterprise Edition - Flexibility Cloud offer
      • How to apply a patch - Enterprise Edition - On Premise offer
    • Where is the Changelog?
    • How to upgrade to a major version?
      • Upgrade from 3.2 to 4.0
      • Upgrade from 4.0 to 5.0
      • Upgrade from 5.0 to 6.0
      • Upgrade from 6.0 to 7.0
  • Import and Export data
    • How import works
    • Understanding the Product Import
    • Understanding the Product Export
    • Formats
      • Localized labels
      • Scopable labels
      • Association types data structure
      • Attribute data structure
      • Category data structure
      • Family data structure
      • Family variant data structure
      • Group data structure
      • Options data structure
      • Product data structure
      • Product model data structure
    • Akeneo Connectors
    • How to Customize Import / Export
      • How to create a new Connector
      • How to import Products from a XML file
      • How to clean a CSV file during a Product import
  • Manipulate the Akeneo PIM data
    • How to Customize Mass Edit Operations
      • How to register a new bulk action
      • How to Register a New Mass Edit Action on Products
    • How to Manipulate Products
      • How to Query Products
      • How to Create Products
      • How to Update Products
      • How to Validate Products
      • How to Save Products
      • How to Remove Products
    • How to Manipulate Non-Product Objects
      • How to Query Non-Product Objects
      • How to Create Non-Product Objects
      • How to Update Non-Product Objects
      • How to Validate Non-Product Objects
      • How to Save Non-Product Objects
      • How to Remove Non-Product Objects
    • How to add a custom action rule
      • General information about rule format
      • How to add a custom action in the rule engine
    • How to Define Access Control List
    • How to Customize the Catalog Structure
      • How to Create a Reference Data
    • How To Customize Teamwork Assistant (Enterprise Edition)
      • Customize notifications
      • Add a calculation step
      • How to log calculation step
      • Remove projects impacted by a custom catalog update
    • How to store assets externally
    • How to Configure Measurement Limits
    • How to Configure the Event Subscription network restrictions
  • Maintain Akeneo PIM projects
    • First aid kit
    • Bug qualification
    • Common issues
    • Scalability Guide
      • Audit with 3 Representative Catalogs
      • More than 10k attributes?
      • More than 10k families?
      • More than 10k categories?
      • More than 500 attributes usable in the product grids?
      • More than 100k products to export?
      • More than 1GB of product media to export?
    • How to purge history
      • How to purge jobs executions
      • How to adapt the version purger to your needs
      • How to purge events API logs
  • Contribute to Akeneo PIM
    • How to report an issue?
    • How to translate the user interface?
    • How to enhance the documentation?
    • How to contribute to a Connector?
    • How to submit a patch to the PIM?
    • How to contribute to the frontend part of the application
    • How behavior tests are architectured in the PIM?
      • Establishing Decorator Pattern
      • Using Spin In Behat
  • Use SSO authentication locally
  • Reference Entities
    • Configure Entity Limits
    • Create a new Reference Entity Attribute type
    • Enrich Records with a new Reference Entity Attribute type
    • Add a Custom Property to Your Custom Attribute Type
    • Refresh records completeness
  • Troubleshooting guide
  • Technical overview
    • Product Information
    • Teamwork Assistant (Enterprise Edition)
      • Project creation
      • Project completeness
      • Project Completeness widget
      • Catalog update impact
      • Scalability guide
      • Users permission summary for Behat tests
    • Collaborative workflow
      • Simple workflow
      • Partial workflow
  • Technical architecture
    • Best Practices
      • Create a project
      • Create a reusable bundle
      • Code Conventions
      • Coding Standards
    • How to implement your business logic using the event system
    • Events
      • Storage events
      • Workflow events (Enterprise Edition only)
    • How to Localize your data
      • How to change the PIM locale
      • How to Use Localizers
      • How to use Presenters
    • How to Add a Notification
    • Performances Guide
      • Memory usage of rules execution (Enterprise Edition)
      • Memory leak fix in Rules Engine (ORM)
      • More than 100 WYSIWYG editors in a page
      • PHP7 and HHVM Compatibility?
      • Job product batch size
    • How to Use the Web REST API
    • Standard format
      • Products
      • Other entities
      • Usage
    • Application Technical Information
      • Application Technical Dependencies
      • Server side set up for hosting
      • System Requirements
      • Recommended configuration
      • Client side configuration and compatibilities
      • Operation processes
      • Flow Matrix
  • Akeneo Cloud Edition
    • Flexibility
      • Environment accesses
      • System Administration & Services Management
      • Periodic tasks & Crontab configuration
      • Composer settings
      • Queue Management & Consumers
      • Disk Usage Management
    • Serenity
  • Akeneo Onboarder
    • Prerequisites
    • How to install the Onboarder bundle
    • Synchronization
    • How to update a minor version or to apply a patch
      • How to update the Onboarder bundle - Enterprise Edition - Flexibility Cloud offer
      • How to update the Onboarder bundle - Enterprise Edition - On Premise offer
    • How to upgrade to a major version
    • Troubleshooting
    • How to uninstall the Onboarder bundle
    • Environment variables
      • Using the DotEnv file
      • Using environment variables
  • About Log4Shell Vulnerability for SaaS Users
  • About Log4Shell Vulnerability for Flexibility v5 Users
  • About Log4Shell Vulnerability for Flexibility users below v5.0
  • About CVE-2022-46157

Setting up the job queue daemon¶

Purpose of the queue¶

Jobs launched from the UI or from the CLI are pushed into queues in order to be processed in background.

One or several daemon processes have to be launched to execute the jobs.

A daemon process can only execute one job at a time. The daemon process cannot execute any other job until the end of the current job. You can launch several daemon processes to execute multiple jobs in parallel.

Also, the daemon processes could be run on several instance of the PIM, using the same MySQL database.

These queues allow horizontal scalability of the PIM. Therefore, you can configure servers dedicated to the execution of the jobs.

Jobs are categorized in three different types and are consumed by three queues (one for each type):

  • ui_job for jobs that are launched specifically by the PIM users (except imports and exports). For instance the mass edit or mass delete jobs.

  • import_export_job for import and export jobs

  • data_maintenance_job for all other jobs that are launched in background

Having these 3 types allows to define a priority between jobs. For instance the following command consumes the UI jobs first, then the import/export jobs, and finally the data maintenance jobs (this is the recommended usage):

1$ /path/to/php /path/to/your/pim/bin/console messenger:consume ui_job import_export_job data_maintenance_job --env=prod

You can also run the daemon and specify how many jobs you want to execute thanks to the limit option. This is useful for development purpose.

1# Run one job then exit
2$ /path/to/php /path/to/your/pim/bin/console messenger:consume ui_job import_export_job data_maintenance_job --env=prod --limit=1

Another possibility is to launch several daemons that will consume or exclude specific job types. This could be useful if for instance too many imports/exports are launched too often. You can add one or a few additional daemons that consume import/export jobs only, and speed up the queue consumption.

1# This daemon will consume import/export jobs only.
2$ /path/to/php /path/to/your/pim/bin/console messenger:consume import_export_job --env=prod

When daemons are running, you can stop them properly by using the following command. The daemons will wait for the end of the current running job (or don’t wait if no job is running) before ending.

1$ /path/to/php /path/to/your/pim/bin/console messenger:stop-workers --env=prod

If the consumers don’t stop after a while, please check that the --env option is the same as the one used to launch the consumer.

Logs¶

The daemon process writes logs to the standard output. Adding a -vv option increases the verbosity and allows to see logs about consumed jobs. It’s your responsibility to choose where to write the logs. For example, to write in the file /tmp/daemon_logs.log:

1$ /path/to/php /path/to/your/pim/bin/console messenger:consume ui_job import_export_job data_maintenance_job --env=prod -vv >/tmp/daemon_logs.log 2>&1

Do note that you should ensure the log rotation as well.

Option #1 - Supervisor¶

It’s strongly recommended to use a Process Control System to launch a daemon in production. This is not useful in development though.

In this documentation, we will describe how to configure the Process Control System supervisor, to run a daemon process. These instructions are valid for Debian 9 and Ubuntu 16.04.

Installing supervisor¶

Install supervisor:

1$ apt update
2$ apt install supervisor

For the other platforms, you can follow the install section of the official documentation.

Configuring supervisor¶

Create a file in the configuration directory of supervisor /etc/supervisor/conf.d.

1[program:akeneo_queue_daemon]
2command=/path/to/php /path/to/your/pim/bin/console messenger:consume ui_job import_export_job data_maintenance_job --env=prod -vv
3autostart=false
4autorestart=true
5stderr_logfile=/var/log/akeneo_daemon.err.log
6stdout_logfile=/var/log/akeneo_daemon.out.log
7user=my_user

The user my_user should be the same as the user to run PHP-FPM.

Then, bring the changes into effect:

1$ supervisorctl reread
2$ supervisorctl update

Launch the daemon¶

1$ supervisorctl start akeneo_queue_daemon

Option #2 - systemd¶

If you prefer, you can use systemd, which allows multiple daemons to run at the same time, log management, and auto restart in case of failure.

As of 3.1, job consumers can be assigned specific job types they will support. This can be leveraged to make sure certain types of jobs will always be processed by a given consumer without being impacted by regular activity on the PIM.

Configuration files¶

Create /etc/systemd/system/pim_job_queue@.service:

 1[Unit]
 2Description=Akeneo PIM Job Queue Service (~/.systemd/pim_job_queue/%i.conf)
 3
 4[Service]
 5Type=forking
 6User=root
 7WorkingDirectory=/path/to/home/user/.systemd
 8ExecStart=/usr/local/bin/pim_job_queue_launcher.sh %i
 9After=apache2.service
10Restart=always
11
12[Install]
13WantedBy=multi-user.target

Create /usr/local/bin/pim_job_queue_launcher.sh:

 1QUEUE_IDENTIFIER=${1}
 2
 3JOB_TYPES=""
 4CONF_FILE=/path/to/home/user/.systemd/pim_job_queue/${QUEUE_IDENTIFIER}.conf
 5
 6if [ -f ${CONF_FILE} ]; then
 7while read job; do
 8    JOB_TYPES+="$job "
 9done <${CONF_FILE}
10fi
11if [ -z "${JOB_TYPES// }" ]; then
12    echo "${CONF_FILE} does not exist or is empty, this consumer will support all job types"
13    JOB_TYPES="ui_job import_export_job data_maintenance_job"
14fi
15
16su -c "/path/to/akeneo/bin/console messenger:consume --env=prod ${JOB_TYPES} &" akeneo
17
18exit 0

At this point, you can create files under /path/to/home/user/.systemd/pim_job_queue/. These files have to be named x.conf, with x being the identifier of the queue, for the sake of this example, the files contain a list of job instance to support, one code per line.

1ui_job
2import_export_job

If the file is empty or does not exist, all job types will be supported by the daemon.

Manage the services¶

 1# use * if you want the operation to apply on all services.
 2systemctl [start|stop|restart|status] pim_job_queue@*
 3
 4# start a pim job queue, configuration in /path/to/home/user/.systemd/pim_job_queue/1.conf
 5systemctl start pim_job_queue@1
 6
 7# start another one, configuration in /path/to/home/user/.systemd/pim_job_queue/2.conf
 8systemctl start pim_job_queue@2
 9
10# check the logs in real time for daemon #2
11journalctl --unit=pim_job_queue@2 -f

Manage services by non-root users¶

sytemctl is not useable by non-privileged users, if you want to allow a user akeneo:

1apt install sudo
2visudo

You can then type in the following lines, depending on what commands you want to allow.

1akeneo ALL=(root) NOPASSWD: /bin/systemctl start pim_job_queue@*
2akeneo ALL=(root) NOPASSWD: /bin/systemctl stop pim_job_queue@*
3akeneo ALL=(root) NOPASSWD: /bin/systemctl status pim_job_queue@*
4akeneo ALL=(root) NOPASSWD: /bin/systemctl restart pim_job_queue@*
5akeneo ALL=(root) NOPASSWD: /bin/systemctl reload pim_job_queue@*

Found a typo or a hole in the documentation and feel like contributing?
Join us on Github!