SCDS1001 - Artificial Intelligence Literacy I

L-5: AI in Physical World

Overview

Physical AI represents a revolutionary leap in artificial intelligence, allowing systems to learn,

adapt, and interact with the physical world without being preprogrammed for every step. Physical
AI systems process data from their sensors, reason about the environment, and take actions
independently. This enables them to handle new, unseen situations, making them far more flexible
and practical in real-world applications.

In 2025, real-world examples include self-driving cars, which operate in select cities by collecting
data, reasoning about traffic, predicting hazards, and making real-time decisions to revolutionize
transportation. Similarly, humanoid robots powered by Physical AI are deployed across industries
such as healthcare and manufacturing, where they adapt to complex tasks, interact with humans,
and perform duties with minimal supervision.

Components of a Physical AI System

A typical Physical AI system consists of several core components that work together to perceive,
reason, and act within the physical world. These components include sensors, actors, and an AI
system (the brain), each playing a critical role in enabling the system to interact effectively with
its environment.

Sensors are devices that collect information from the physical world, such as light, sound,
temperature, or movement, and convert it into digital data that the AI system can understand. This
sensory data forms the foundation for the AI’s perception and decision-making processes. In
autonomous vehicles, for example, multiple types of sensors are used to ensure reliable
performance. LiDAR uses laser pulses to create detailed 3D maps of the environment, making it
highly effective for detecting objects and measuring distances with exceptional precision. Cameras
capture visual data, such as images and videos, to identify objects, lane markings, and traffic signs,
which are essential for interpreting fine visual details in the surroundings. Radar, on the other hand,
uses radio waves to detect objects and measure their speed and distance, and it is particularly useful
in adverse weather conditions like fog, rain, or dust, where its ability to penetrate obstructions
becomes invaluable.

Sensor fusion technology is a crucial component that combines data from multiple sensors to
create a comprehensive and accurate understanding of the environment. By merging information
from devices such as LiDAR, cameras, and radar, the system can overcome individual sensor
limitations and deliver reliable performance in a variety of conditions. For instance, in autonomous
vehicles, sensor fusion ensures accurate object detection and situational awareness, even in
challenging scenarios like heavy rain or low visibility. This integration of multiple data streams
allows Physical AI systems to operate safely and effectively in complex real-world environments.
By combining these components—sensors, actors, and sensor fusion—Physical AI systems are
equipped to perceive their surroundings, reason about what they observe, and take actions that
enable them to function seamlessly in the dynamic physical world.
Actors are devices or mechanisms that execute physical actions based on the decisions made by
the AI system. These actions allow the system to interact with its environment and achieve its
objectives. For example, a robotic arm may move to pick up or manipulate objects, or a motor in
a self-driving car may adjust to steer, accelerate, or brake. Actors can also activate lights, turn
appliances on or off, or make other physical adjustments in response to environmental changes.
Essentially, actors serve as the “hands and feet” of a Physical AI system, translating AI-driven
decisions into meaningful physical actions.

The intelligence behind Physical AI systems lies in their ability to perceive, reason, and make
decisions. Perception involves transitioning from raw data collection to meaningful interpretation,
such as using cameras and LiDAR to detect objects and identify pedestrians, vehicles, or obstacles.
Reasoning allows the system to build a contextual understanding of the environment, like
predicting that a pedestrian might cross the road based on their movement and position. Finally,
decision-making enables the system to predict future events, simulate possible outcomes, and
formulate strategies to act—for example, an autonomous car deciding to slow down or change
lanes to avoid a potential collision. Together, these capabilities make Physical AI systems adaptive
and effective in dynamic real-world scenarios.

From Seeing to Thinking: How AI Moves from Perception to Reasoning

Computer vision is like the "eyes" of AI—it’s the first step that helps AI "see" and understand the
world. Just like humans look at objects, recognize faces, or read signs, computer vision allows AI
to process images or videos and figure out what’s in them. For example, it can help a self-driving
car recognize pedestrians, traffic lights, or other vehicles on the road. Essentially, it’s the
technology that helps AI make sense of what it "sees" in the physical world.

The core idea behind the Convolutional neural network (CNN), which powers computer vision,
is inspired by how our brains process images. Instead of looking at the whole image all at once, a
CNN breaks it into smaller parts (like scanning tiles of a grid) and looks for patterns in each part.
For example, it might first find edges, shapes, or colors in small areas, and then combine all that
information to understand the bigger picture—like recognizing that those shapes form a person or
a car. It’s like building an understanding layer by layer, starting from simple details and working
up to complex objects. This step-by-step process makes CNNs very good at recognizing and
interpreting images.

AI’s ability to move from perception to reasoning marks a major leap in intelligence. Perception
is about recognizing and interpreting what the AI “sees,” like identifying objects in an image using
technologies such as CNNs. For example, in 2015, AI surpassed human performance in image
classification, where it could correctly categorize images more accurately than humans. But
reasoning takes things further - it’s when AI starts to make sense of what it perceives and uses that
understanding to think or make decisions. A breakthrough in this shift happened in 2016 when
AlphaGo, powered by a 12-layer CNN and trained using reinforcement learning, defeated one of
the best Go players in the world. AlphaGo not only perceived the state of the board but also
reasoned about strategies and long-term consequences of its moves.
By 2020, AI had advanced significantly, even surpassing human performance in Visual
Question Answering (VQA) tasks. VQA refers to the ability of AI systems to answer questions
about images by analyzing their visual content. For example, in a VQA task, an AI might
identify objects in a picture, such as "a cat" and "a toilet paper roll," and answer factual questions
like "What is next to the cat?" or "What color is the toilet paper?" This requires the system to
combine visual recognition with natural language processing to provide accurate answers based
on the image.

However, as of 2023, AI systems still face greater challenges in more complex tasks, such as
Visual Commonsense Reasoning (VCR). Unlike VQA, which focuses on answering direct or
factual questions, VCR involves making logical inferences about everyday situations depicted in
images. For instance, given an image of a cat sitting next to a toilet paper roll, a VCR task might
ask, "What is the cat likely to do next?" (e.g., "The cat might play with or unravel the toilet
paper") or "Why is the toilet paper at risk?" (e.g., "The cat could knock it over or tear it apart
while playing"). These tasks require not just recognizing objects and their relationships but also
applying real-world knowledge and reasoning to infer intentions, predict outcomes, or explain
situational dynamics. While AI systems have not yet surpassed human performance in VCR,
they are steadily improving, demonstrating progress in their ability to reason about visual
contexts.

Vision-Language Models (VLMs) combine visual understanding with language reasoning. In

simple terms, a VLM can “look” at a picture, “read” or process the image, and then connect it to
written or spoken language. For example, if you show a VLM an image of a cat with a roll of toilet
paper, it can reason that the cat might have unrolled the paper because it recognizes the scene and
understands the typical behavior of cats. This ability to combine vision and language helps AI not
just see or describe the world but also understand it more deeply.

How Does AI Learn to Make Decisions? Simulation and Reinforcement Learning

AI learns to make decisions by practicing in controlled environments where it can safely

experiment, make mistakes, and improve. One of the most important tools for this learning process
is simulation. Simulations create virtual worlds where AI can train without the risks or limitations
of the real world. These virtual environments allow for precise control over factors like friction,
gravity, or lighting, which makes it possible to tailor training for specific tasks. For example, in
robotics, simulations let robots practice handling objects or navigating spaces without risking
hardware damage or safety concerns.

In 2025, Nvidia announced a new simulation tool to accelerate the training of autonomous vehicles.
This tool generates synthetic driving scenarios for AI, such as handling icy roads, sudden traffic
changes, or low visibility conditions. These scenarios help self-driving cars practice decision-
making in a wide variety of situations that would be difficult—or even dangerous—to recreate in
the real world. Simulations like these are critical for preparing AI to handle real-world
complexities.

Another key method for teaching AI to make decisions is Reinforcement Learning (RL). Think
of RL as teaching AI to learn through trial and error, similar to how people learn new skills. In RL,
the AI (called the agent) interacts with its environment and learns by receiving rewards for good
actions and penalties for bad ones. Over time, the AI tries to maximize its rewards, gradually
figuring out the best decisions to make in different situations.

Reinforcement Learning allows AI to learn from its mistakes and improve without needing explicit
instructions. For example, in a driving simulation, the AI might initially make poor decisions and
crash, but over time it learns to avoid those mistakes by understanding what actions lead to better
outcomes. This ability to adapt and improve makes RL a powerful tool for teaching AI to make
decisions in complex, dynamic environments.

Physical AI: Where Are We and What’s Next?

Self-driving technology, one of the most advanced examples of Physical AI, is progressing through
six levels of automation. These range from no automation at all (Level 0), where the driver is fully
in control, to full automation (Level 5), where the vehicle can drive itself in any condition without
human input. Currently, we are at Levels 3 and 4. Level 3 vehicles can take full control of driving
in specific conditions, such as on highways, but require the driver to intervene when needed. Level
4 vehicles go further by handling all driving tasks autonomously in restricted areas or conditions.
However, fully autonomous Level 5 systems, capable of operating in all scenarios without human
involvement, remain a distant goal.

Despite the technology being ready for Level 3, adoption has been slow due to factors beyond
technical readiness. For instance, many regions do not permit drivers to take their eyes off the road
even if the vehicle is capable of driving itself. Legal uncertainties also play a role, as liability
remains unclear in the event of an accident—should the driver or the manufacturer be held
responsible? Beyond these regulatory and legal challenges, societal factors such as cultural
attitudes, public trust, and education also impact adoption. Many people remain hesitant to trust
AI-driven systems, and concerns about safety, potential job losses, and system failures contribute
to slower acceptance.

Looking ahead, the future of Physical AI will require more than technological improvements.
While advancements in AI decision-making and adaptability will drive progress toward higher
levels of automation, governments, manufacturers, and communities must work together to
address regulatory gaps, build societal trust, and educate the public. Only by overcoming these
challenges can we fully unlock the potential of autonomous systems and take the next step in
Physical AI innovation.

Ending

“As HKU students, you are innovators, tech pioneers, policy shapers, and changemakers. The
future of AI lies in your hands, carrying with it the responsibility to guide it toward the
betterment of society. Trust in your vision - the world is waiting for you to lead and make an
impact.”

Kit