AI Integration In Business Operations: Regulations, Risks, And Ethical Considerations

Insights from the EU AI Act

Businesses increasingly rely on technology containing artificial intelligence (AI) to streamline operations, enhance employee experiences and drive strategic decision-making. The integration of AI into work systems, including human resources information systems (HRIS) and other platforms, presents both unprecedented opportunities and challenges, though. 

As companies embrace AI, it is crucial to navigate legal and ethical considerations effectively. This article provides actionable insights for companies to evaluate AI systems.

Regulations

An important place to start is the current and anticipated regulatory framework. On February 13, 2024, the European Parliament’s Internal Market and Consumer Protection Committee adopted the AI Act. It will be submitted for a plenary vote that is provisionally scheduled for April 10-11, 2024. The Act is the world's first comprehensive law on artificial intelligence.

Various U.S. local and state governments have also passed and are passing legislation to govern the use of AI tools in employment, and the White House Office of Science and Technology Policy has issued the Blueprint for an AI Bill of Rights. The focus of the laws, proposed regulations and the Bill of Rights is on maintaining safe systems, ensuring data privacy, offering notice and explanation of the use of any automated systems, and providing opt-outs.

While the EU AI Act or the European Union AI Act may not apply to all U.S. companies and will not be enforced until 2025, it provides an excellent overarching framework and guide to what companies can expect to see in future AI regulation. Also, any company doing business in the EU should ensure compliance with the Act because once enforceable, companies that violate the EU AI Act’s rules could face fines of up to 35 million euros or between 1.5% to 7% of their global sales in the preceding financial year.

What Is AI?

To comply with the act, a company must first identify the presence of AI in its systems. The publicly available EU AI Act Compliance Checker is a good resource.

The EU AI Act provides a good definition of AI, which follows the Organization for Economic Co-Operation and Development’s latest definition of an AI system as “a machine-based system designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.”

Risk Management

After identifying the presence of AI, a company must understand the level of risk associated with the use of such technology. The EU AI Act follows a risk-based approach to regulation, categorizing AI applications into four levels. The higher the risk, the stricter the governance.

The levels include (1) Unacceptable risk: A system that poses a threat to humans, such as cognitive manipulation; (2) High risk: A system that affects the safety of individuals and fundamental rights, such as credit scoring systems and automated insurance claims; (3) Limited risk: AI tools like chatbots; (4) Minimal risk: Applications such as video games or spam filters. 

AI systems identified as high-risk include AI technology used in employment, management of workers and access to self-employment, such as an HRIS system. Other high-risk systems would include critical infrastructure, education and vocational training, essential private and public services (e.g. healthcare, banking, etc.), certain systems in law enforcement, migration and border management, and justice and democratic processes (e.g. influencing elections).

Any company that is considering procuring high-risk AI should establish a risk management system (including a record keeping component). Such a system should include a plan to conduct data governance and ensure that training, validation and testing datasets are relevant, sufficiently representative and, to the best extent possible, free of errors and complete according to the intended purpose. Any such program should include human oversight to establish a quality management system to ensure compliance.

Key Questions

In addition to any internal steps for the use of AI, a company seeking to use a vendor selling AI (such as an HRIS) should ask the vendor whether they have taken the required steps to comply with the EU AI Act.

Some examples of key questions to consider asking:
 

• How do your AI systems comply with relevant regulations, such as GDPR (General Data Protection Regulation), CCPA (California Consumer Privacy Act), or industry-specific standards?
• How do you address ethical considerations and ensure compliance with emerging AI regulations, such as the EU AI Act or similar legislation?
• What measures are in place to ensure data privacy and security?
• Can you explain your approach to data encryption, access controls, and data breach prevention?
• How do you handle return of data at the conclusion or termination of a contractual relationship?
• How do you mitigate biases in your AI algorithms?
• What measures do you take to ensure accuracy of AI predictions, recommendations, or decisions?
• How do you improve the performance of your AI systems over time and monitor performance?
• Do you offer support and training to assist companies in implementing and using your AI systems effectively?
• What measures do you take to ensure your AI solutions remain adaptable to future technological advancements?

When selecting AI system vendors, it is important to conduct due diligence to assess their compliance efforts and commitment to ethical AI practices.

Key Considerations

In general, key considerations include: (1) ensuring the data stored is encrypted; (2) inquiring about what the vendor has done to comply with the EU AI Act; (3) obtaining explicit consent from employees before collecting and processing their personal data; (4) ensuring vendor agreement includes timeline and requirement that any personal data is returned to the company upon termination of such agreement; and (5) providing training on any new systems.

This article first appeared on the Fox Rothschild website.

Bryn Goodman is a Partner at Fox Rothschild.